Most of them are fine - it's just a small handful, like Pauline, Giga Mac, 9-Volt and 18-Volt, Akuma, and Dr. Wily, that are extremely difficult.
That is only if you have the counters at least though.
In a neutral setting you still get bodied, because of the hard AI.
Using Legendary Spirits, with broken Supports (Super Armor and things like that) is the only consistent way to win most of the Spirit Battles.
The artificial difficulty comes from the ground hazards, especially if they cover the whole stage and not just the bottom platform (which the description doesn't differentiate between btw) and thus force you to abandon the very useful abilities.
...and the other thing, is you being outnumbered. Which is just unfair.
And unlike in past Smash games, the team that outnumbers you doesn't have their defense and AI lowered to compensate,
Those two are just the biggest things that break the whole thing for me personally.