As someone who used to be embarrassingly anti-competitive myself, I can answer this.
• Casual and competitive players find very different aspects about the game appealing, to the point of seeing them as entirely different genres. Competitive players tend to be in it for the fighting, and casual players tend to be in it for the colorful cast and cathartic feedback. They don't see it as a test of skill, they see it as something quick to do when you have a bunch of friends over.
• Randomness to casual players is a plus, not a minus. They may not have the time, energy, will, or even desire to improve their skills, so games with a randomness factor appeal to them because the match isn't... over before it even starts, so to speak. With items and stage hazards, they have at least somewhat of a fighting chance to come out on top even if they're a newcomer against a very experienced player. In their eyes, if they need to train and study to not be humiliated in something they just want to do to relieve stress for a few minutes, why even bother playing at all? This is the specific factor that Sakurai was worried about the most in the period between Melee and Brawl. Contrary to popular belief, he has absolutely nothing against competitive players—during the Wii U era, after watching a tournament, he basically said he respects the players' skill, but playing that way just isn't for him personally—but he did feel that an insurmountable skill gap prevented the majority of the audience from being able to have fun at all. Like a major league sports team full of hulking professional athletes on steroids challenging an amateur sports team of middle schoolers just trying to enjoy their sunny Saturday in the park—the "steroids" in this case being advanced techniques and combos that do require strength, endurance, cunning, and reflexes well above what the average person can do without intense exercise on the players' part. It's not about "getting good" for these players—in situations like these, they are literally physically incapable of not being crushed.
• Many casual players feel it's disrespectful and insulting to the spirit of the franchise, twisting it into something it was never meant to be.
• Many casual players are very, very irritated by repetition, so the relatively limited pool of stages, characters, and music tends to grate on their nerves. (For me when I was like this, it was the Dream Land 64 theme specifically that I basically went "If I have to hear that damn song even one more time--!" with.)
• They feel that items and stage hazards are a significant part of the experience and to ignore them is at best a waste of the game.
• They see the idea of tier lists and stage lists as artificial gatekeeping by people who just want to police everyone else's experience. "Tryhards" who blindly follow stage lists/tier lists like scripture, demanding everyone else do the same, without actually knowing what they're for have tainted their experience in this regard further and basically given them evidence to strengthen the misconception.
• Bad blood left over from the Brawl days. The Melee fandom actually was incredibly toxic in the late 2000s, and many associate conpetitive play as a whole with the jerks who were supporting competitive play then. (I'd argue the shoe is on the other foot now and it's the casual fanbase that's become ludicrously toxic and intolerant in modern times, but I digress.)
• Those who speak out against GameCube controllers see it as a refusal to accept change, seeing us as no different from old men who insist that games stopped being good after the Atari 2600. "Nostalgia-blind" is a word thrown around a lot—by holding onto the past, to them, we're completely ignoring progress. (And to play devil's advocate for a moment, they do have a point about the GC controller being outdated; most modern games cannot be played with it due to having several buttons missing compared to any other modern controller—namely, L1/LB, Select/Minus/Back, L3/LSB, R3/RSB, and PS/Guide/Home. I would suggest a third-party GC-shaped Pro Controller like the ones HORI makes instead for this reason.)
As much as Brawl sacrificed from the conpetitive crowd, it filled the casual niche flawlessly with its forgiving nature that makes people who mess up not feel like it's hopeless to even try to make a comeback, and big, flashy, high-feedback hits that sent enemies far away and just felt good to land. There's a reason that Wii U/3DS wasn't NEARLY as well-recieved even by casual players. There's a reason characters like Ganondorf are so popular casually—for them, feedback is everything, and with most fighting games, they see individual attacks as being relatively light and dinky with barely any effect on the opponent and, thus, rather boring to see and do.