Flash back to 2001: Melee was released. Everyone loved it, casuals (which was pretty much all that existed at the time) and hardcore alike. There was something for everyone. The mechanics were responsive and solid, there were skill based elements along with some random elements, but it was all optional so people could tailor their experiences however they wanted. As the years went on and the competitive aspects of the game developed, it was clear there were plenty people who truly enjoyed the depth that Melee had to offer, enough so to remove a lot of the less than controllable elements of the game and put money and pride on the line at tournaments. There were few that could argue that Melee had the best of both worlds.
Flash forward to 2006: The first trailer for Brawl was shown. There was nothing before, and few things since, that have gotten me hype for a game. New characters, new stages plenty of "OH **** DID METAKNIGHT JUST WAVEDASH WTF WAS THAT WHO THE **** IS PIT OMG WARIO WTF SOLID SNAKE?" Everyone was super excited for it. Everyone thought it would be an improvement on Melee in every way. Smash players at that time WERE MELEE PLAYERS and that didn't stop them from being excited about Brawl. The hate for Brawl did NOT come out of nowhere. My point here is, Melee players are not inherently against Brawl because it's Brawl. If you choose to be ignorant and assume that Melee players simply can't "adapt" or are simply choosing to dislike the game because they are "tryhards", then take your moronic arguments elsewhere.
There was no "this town ain't big enough for the two of us" situation. That is why many competitive players felt betrayed by Sakurai. The less competitive crowd was more than happy with Melee how it was, and the competitive crowd wanted the competitive aspects expanded rather than diminished. You can argue all day whether or not Brawl is a competitively deep game, and I don't pretend to be an expert on the matter, but what you can't argue with is the fact that, in terms of gameplay, Brawl lacks many fundamental aspects that, whether or not you'd say they required more / added more depth to the game, at the very least created a larger skill gap between people who could perform these techniques and people who could not.
A game with depth is a game with depth. That is what Smash is; whether it is technically a fighting game or not (as determined by who? Not sure. Doesn't really matter) is irrelevant. It functions similarly to a fighting game minus some major elements (health bar, meter, etc.) but has some unique elements (platforms, much more freedom of movement, direction influence, etc.) and it is definitely not for everyone.
The overly complex ruleset is there for the exact reason that everyone knows already: In vanilla settings, the game is not fit for competitive play. That is because it was not designed to be that way (by default). One thing that everyone has to realize is that regardless of a games intended design, there can be unexpected results. Melee's depth exists whether intentional or not, and there is nothing wrong with people taking their enjoyment of the game to another level and attempting to mimmick "real" fighting games. Anyone who is annoyed or upset by this needs to get over it. If it weren't an enjoyable experience then the Smash community would be dead by now, and it isn't.
Now if I'm understanding the OP correctly, your saying that the new Smash will be neither like Brawl nor Melee? I suppose that is a strong possibility, but here is the ultimate truth: Was Brawl a market failure due to the fact that the hardcore gamers disliked the changes? No, not in the slightest. The fact of the matter is that Nintendo doesn't really care how much we love Melee. They could make the next Smash the most competitive Smash to date, and the effects on it's sales would be pretty minor because competitive Smash players are by and large the minority. I'd be extremely thrilled if the next Smash was brought back in line with what made Melee so amazing but I'm not holding my breathe.