I think when people keep saying that a game can be fun casually and competitively, it's a bit more complicated than originally perceived. I can see where Samurai is concerned, and that it's not total BS like many people claim it is. I first noticed this when I was playing with some of my casual friends.
They all said something with the same general idea behind it: playing Melee with me wasn't fun because getting destroyed and overwhelmed with Falco/Fox pressure felt bad. They felt that at least in Brawl, even though I was still fundamentally better than them, the difference in skill wasn't felt nearly as much, and they felt that they still had a chance, etc.
As much as it is tempting to just tell them to play more, get better and learn how to deal with the pressure, one must understand that everyone is different, and not everyone wants to play and learn at such a serious level; they just want to mess around.
This is where I think the problem lies: when the skill ceiling is high for a game, it automatically deters casual players from playing the game, especially in a primarily local game where the casual and competitive groups often mix. I feel like having a good matchmaking system may help fix this issue, where casual players will play other casual players, and vice versa for competitive players. This has to be done well though, as I, for example, hate being able to see my own statistics; I only want to play more and get better, no matter what the result is, and I find it all too easy for myself to get caught up in statistics.