So we basically have to figure out which series became massive successes in this timeline. Honestly the best idea I've seen for "hypothetical Smash 64 but thing" threads. Congratulations.
I'm guessing Super Kid Icarus was made in place of Super Metroid.
Punch-Out games became bigger and bigger. Punch-Out 64 was one of the best-selling launch games of all time.
Speaking of N64 launch titles, the N64 would never have been successful if it wasn't for the ambitious Ice Climber 64. This game set the industry standard for 3D games coming out in the next few decades, with its jaw-droppiong use of the vertical axis for exploring vast 3D mountains, and its innovative real time multiplayer exploration system.
Banjo-Kazooie, influenced by the success of Ice Climber 64, also developed its own multiplayer exploration system and came out in 1998 as a slightly different game as we know it, but to the same amount of critical acclaim.
And of course, you can't speak of Rare without speaking of the classic arcade game reimagined as a 2D platformer trilogy for the SNES, featuring state-of-the-art pre-rendered silicon graphics: Wild Gunman Country, released in 1994. After this, Nintendo made sure to buy Rare, officially making them and their series part of their first-party lineup of games.
After the critical and commercial failure of Star Fox and the FX chip in the SNES, Nintendo decided to take another jab at the 3D arcade dogfighting concept in more powerful hardware. Space Firebird 64 was a revival of their arcade classic, and with its exciting setpieces, fully voiced dialogue, cheesy but massively entertaining story and complex branching paths, it was destined for success from the start.
The N64 was a great system, but it wouldn't have the success it had with casual audiences if it wasn't for Balloon Fight Kart and Balloon Fight Party. Balloon Fight was becoming a worldwide phenomenon, and it would be hard to imagine a world where Balloon Fight games didn't release multiple times per year, but it's games like this that truly made them a part of pop culture. The fact that there wasn't a main-series Balloon Fight game for the entire N64 generation is still a mystery, but all these other games made a decent job to keep fans happy until their grand return on the Nintendo Gamecube.
You can't make a Nintendo system without a new version of the Robotic Operation Buddy, more commonly known as R.O.B. The N64 features R.O.B. III, and its complex artificial intelligence can learn to play just about every multiplayer N64 game out there. While you can use him as a partner for games like Ice Climber 64, Banjo-Kazooie or even Smash itself, there's no game that takes advantage of him as much as Gyromite 64.
Also they made another ****ing Mach Rider, my imagination is running out.
In this timeline, there's no doubt that the roster of fighters would be something like this:
-Popo (Ice Climber)
-Pit (Kid Icarus)
-Little Mac (Punch-Out!)
-Banjo (Banjo-Kazooie)
-Wild Gunman (Wild Gunman)
-Firebird (Space Fire Bird)
-Balloon Fighter (Balloon Fight)
-R.O.B. III. (Nintendo)
And the four unlockable fighters, sharing some of the animations of the 8 originals to different extents:
-Nana (Ice Climber)
-Kazooie (Banjo-Kazooie)
-Professor Hector. (Gyromite)
-Mach-Rider.