So it's more balanced if you don't have any friends for the social stuff? Because I never bothered with friendlist stuff for 3ds, so I have no friends to summon.
I feel that way, for the most part, yeah. Some might argue that if you get a friend summon like this and use it, for example:

...because they know that using Bravely Second amps your attack like crazy and they're in postgame or late-game, then you can finish things off no problem, provided if you use it (and you definitely can decide not to use them). I didn't use friend summons unless they were quick heal/buffs because a lot of people I knew had stronger characters I knew that these would wipe out some of the side or main story bosses easily. You can even eliminate the secret boss with that, no problem.
Additionally, you're going to get access to a lot of their advanced skills in the classes that you have, provided you're using Abilinks and assigning some of those advanced skills to your support abilities. If you assign some of the jobs as primary and secondary and have Abilinks turned on to those who are late/post-game and have basically 100%'d everything, then you also get access to their active/passive abilities per job before you probably 'should'. It makes the game much easier at that point in time, especially if you just got access to said job. I
do think having some Abilinks can be neat if you want some actives for a subjob, so it entirely depends on your judgement.
I don't think it
ruins the game, per se, but it's something that I started to realize when I played For the Sequel, and scaled back on using because while it's cool to have these community aspects, I wasn't seeing how the base game was balanced provided I was playing myself. I didn't streetpass anyone because I live in NA and no one I knew imported the JP version so I built up my village on my own, too.
Otherwise, there are lots and lots of other ways to break the game, too. Using all Valkyries and giving them a ton of speed abilities + someone with Hasten World so you'll never ever be attacked during a battle while using Jump. Using Swordmaster/Ninja combos, etc.
However, I think the social stuff is beneficial because it does play a minor role in the game's narrative. I thought it was one of the neatest things given the overall narrative premise of the game.
I still think the odd balancing here and there isn't the biggest flaw the game has. It's really the crappy pacing in the second half.