Characters aren't the be-all / end-all of the Smash experience, but they're a pretty big part of it. That much I daresay everybody can acknowledge.
That being said, if you like other elements of the game beyond the characters, good for you. There's plenty of nice things in the game to be appreciated, from the stages to the music, even the items and the trophies. Everybody's opinion is valid in that respect. We all play for our own personal reasons and that is okay.
Delightfully phrased. Bravo.
I think there are a couple reasons people get so worked up over characters. For one, they are, arguably, the biggest part of the game. They're what you are actually in control of, they're the extension of the physical players. We've come to expect a huge variety, and it's easy to lose sight and be very let down by characters you don't immediately understand, frustrate you, or are just missing and people really wanted. They're such a massive piece of the definition of the game.
Additionally, they're probably the longest lasting element of the game. All the different game modes and the joys those bring eventually subside into casual party play and competitive play, both of which revolve around the basic gameplay elements: the characters, the stages, and the items. A huge portion of the community only uses items occasionally for fun (or not at all,) and stages have varied acceptance, but even with that a seldom few of the total are accepted by the majority of lasting players. The characters, however, are what remain through its lifetime, and, as such, are a massive part of what makes the game what it is.
Certainly, the characters aren't everything, but they're such a massive portion, and to the people who they really matter to, it's not helpful to label them and tell them their opinion of the game is wrong. Everybody is free to like or dislike the game for their own reasons, as long as everybody can see that then there is no problem to be had.