Before I begin, I want to say I really like this comment. This is the kind of discussion I want. Kudos to you.
You have a good point, but where I disagree with that I think you are looking in a theoretical sense rather than what is actually going on. I'll summarize why
- Unlike Brawl and Smash 4, there is no one popular guest character
- Nintendo is producing more IPs than they have in the past. Since Splatoon, Pikmin was the last IP which featured characters and worlds. Nintendo has made other IPs, but they were usually like Wii Sports or the Touch Generation games. Nintendo is also making ARMS and will probably have more IPs at E3 given how they are talking (speculation on my part).
- With guest characters, we could be reaching negative utility. Although some have claimed the "character I want to be cut" for Cloud and Bayo are due to them being good, the same wasn't true of other good characters besides Bayo. There was still plenty of controversies when the characters were announced
On the guest character, I think its fair to say that not all guest are created equally. Snake was a character we expected to be cut, and most people didn't seem to mind (it's only after he's gone that fans actually say "What, we love Snake now"). It's not a surprise that Sonic and Megaman were the most popular characters. Megaman has only been successful on Nintendo systems and Sonic games sell the best on Nintendo systems. This also explains why Cloud was controversial. He's doesn't have the same connection as Megaman and Sonic do.
You mention too many Nintendo series being niche and unsuccessful, but let's look at the third party series. First, Sakurai is Japanese and is a big name in that country. He doesn't speak English, so there is a barrier to working with western companies (and he wants to communicate with the creators). Regardless, most western third party games don't do that well on Nintendo system. Outside of Overwatch, there doesn't seem to be a lot of overlap for Nintendo fans and most western series (but feel free to disagree). So let's focus on Japan for now.
Tekken isn't very popular on Nintendo systems and the series is on the decline. Dragon Quest is unpopular in America. YokaiWatch is VERY unpopular in America. Bomberman games don't sell that well. Most of Sega's series are in the ditch as are Konami's. The only Japanese series that does well worldwide and is popular with Nintendo fans seems to be Monster Hunter. So in terms of picking the best character, there's not a lot here. And again, after doing Cloud, these characters character going to be people excited. Moreover, there are only so many mega iconic characters out there.
But also consider your audience. You mention that there would be diminishing returns for Nintendo characters, and that is true of new characters regardless. The jump from 12 to 25 was bigger than 35 to 55. But if the thought is Nintendo will run out of characters, I direct you to Blizzard's all star game, Heroes of the Storm.
The game has more characters than Smash (60+) from only 4 series. Nintendo has a plethora of series to use. "But these characters are obscure." Not to your target audience. The fans of Smash Bros know Nintendo. This is why people aren't usually upset over new characters unless they are a clone or if there is too much from on series. Conversely, we have seen fans be upset if the guest character doesn't fit the mold.
Perhaps you could argue my suggestions too extreme. OK. Take one guest character from the last two games and add one more. Make them feel special and different. This is another way to reach the same goal.
I will say that in terms an ONLY Nintendo game, you have to remember that Nintendo is the point of Smash Bros. Akuma isn't symbolic of Tekken. Predator isn't symbolic of Mortal Kombat. You mention there won't be a Nintendo character as big as Sonic or Cloud, but there also isn't a guest character that would be as big as Cloud or Sonic. Those days are gone. If there isn't a new guest character fans can get excited over, does it need to continue? I would argue no. Again, the point of Smash is Nintendo, so moving in that direction can never be moving backward. You are never moving away from makes Smash, Smash.
Thanks for taking the time to respond to this - there's a lot to cover here, and I probably won't be able to go over it all, but I'll do my best.
As far as Heroes of the Storm, there seems to be a big difference in terms of scale. I'm not overly into Heroes of the Storm (I play Co-Op against the AIs now and then, but I avoid playing online against random people and don't follow the competitive scene), but by my understanding, characters in that tend to involve a lot less than Smash in terms of moves.
There's a basic attack, three specials, and two ultimates that you can pick between once you reach a certain level. Upgrade trees can branch characters out in a bunch of different ways, but there's only six specific attacks in the game. Smash has way more than that; with K. Rool, it might be easy to think of his B moves and final smash, but it seems to be harder to make his basic attacks interesting. And that's the final boss of a major franchise; sure, attacks can be generic for characters, but I feel like they'd want to avoid making it feel like certain characters blend together in terms of what they can do.
For guest characters in other games, they may not be the central focus of the game, but they can get a game attention from people who otherwise might not have considered it. The two games that you listed are perfect examples. I'm not a huge fan of Mortal Kombat (I got the first three Super NES games as hand-me-downs, but that's about it), but finding out that the Predator was a playable character and seeing him in action got me interested. Same thing goes for Tekken - I only got into the series through Tekken Tag Tournament for the Wii U (wanting to see what the franchise was about, and the Nintendo costumes exclusive to that version, helped), and while I don't know much about the new characters added in Tekken 7 (aside from Kazuya's mother), the addition of
Akuma of all characters made me take notice of it.
Even if they can't top the excitement of Sonic and Cloud, guest characters still have a lot of benefits. For third party companies, they can draw attention to lesser-known franchises like Bayonetta, or revitalize franchises like Mega Man. They also serve to draw new fans into Smash Bros., and Nintendo's series by extension. I touched on this previously, but if you were to make a Venn Diagram of Nintendo fans and Donkey Kong fans, the DK circle would basically be inside the Nintendo one (aside from people who like Nintendo games but don't like Donkey Kong for whatever reason).
If Dixie or K. Rool were to be added, the main people who would be excited would be people who are (in all likelihood) already fans of Smash Bros. Meanwhile, if Crash Bandicoot was added, it would get the attention of people who played the original Crash Bandicoot trilogy years ago and are nostalgic, or people who want Crash Bandicoot to get new games, or people who really like Crash Team Racing (which, by my understanding, is a solid racing game, though I've never actually played it).
I'm not very familiar with Yokai Watch, but having a Dragon Quest character (ideally, one of the iconic monsters, since I'd be concerned that one of the heroes would be a bit generic or close to a Fire Emblem character) playable in Smash would help to give it more exposure to Western audiences, and having someone like Heihachi in Smash could serve to introduce Tekken to plenty of Nintendo fans, like how having Snake in Smash introduced plenty of Nintendo fans to Metal Gear. (myself included)
Alternatively, Square-Enix could go with someone like Crono from Chrono Trigger; while it's a series that only has two games, the original is
extremely popular, especially among Nintendo fans. Rayman definitely seems like a possibility seeing as he already got a trophy in Smash Bros., Ubisoft seems to be up for it, and Nintendo would presumably be okay with it.
I feel like all of the third party characters, with the possible exception of Cloud, are quite special in terms of gameplay, and removing any of them would be a huge loss. Smash Bros. is and always will be focused on Nintendo, but the guest characters add to the fun. It allows us to create scenarios that couldn't possibly happen otherwise, like Link vs. Cloud or Kirby vs. Mega Man, and it allows Nintendo to celebrate some of the developers that have given them support over the years, in addition to celebrating their own history. In all likelihood, Nintendo couldn't have gotten to where it is without the support of companies like Capcom or Namco or Konami, or even Sega (as odd as that still is to say), so why not include them in the festivities?
...This got way longer than I intended it to; sorry about that. I guess it's just something I'm passionate about.