well if you're constantly comboing him and he's barely able to fight back like your post implies, of course he's going to get frustrated and quit, not everyone finds slamming their heads against a wall enjoyable. although the manner in which your brother handled it is not optimal in the slightest
however, i dont know enough about your situation. does your brother like playing PM? what kind of game does he like to play (1v1, free-for-all with items, etc.)? does he have a passion to improve, or does he just like playing PM as-is? why does he prefer the wiimote over other control options (assuming he prefers it)? have you shown him how to do advanced techniques, and has he implemented any of them into his gameplay?
the whole "casual vs competitve" mindset doesn't really exist in a game where the entire point is for many forces to compete against each other for victory. rather, these buzzwords tend to be used to refer to skill levels that people have, which and be both demeaning and inaccurate. generally, people use the word "competitive" to mean "having the drive to compete" rather than just "to compete." for example, if there is a person who knows nothing about wavedashing or l-canceling and is generally not very good at the game, but still strives to better himself by seeking out ways to improve and fighting other players, i would consider this person as being "competitive," as in "having the drive to compete."
to answer the initial question, i think your brother's argument went something like this: "To be competitive, you must attend a tournament. You have never entered a tournament. Therefore, you are not competitive." however, going by my definition of "competitive" that i outlined, attending a tournament is NOT a prerequisite to "having the drive to compete." in fact, i'd argue that even entering a tournament requires a degree of competitiveness, since the entire point of tournaments is for many people around the world to compete against each other, so it's safe to assume that everyone playing in the tournament are already competitive to begin with.
this is besides the point, but being that you two are brothers, it might be best to resolve the differences that you two have in a positive way. assuming that you two grew up together and shared similar experiences, it would be a shame if you two separated under petty conditions. having another person who grew up alongside you and someone that can provide you support is a luxury not many people have. either way, i really shouldnt be telling you how to treat your brother, as it's none of my business