Also, @
LIQUID12A
is right; Ridley appears frequently and is a significant player in Samus's backstory, but as a whole is never really the driving force of it all. I think you could make an argument for him being a central villain in Super Metroid despite not being the final boss (since he initiates the plot by stealing the metroid larva), but other than that he's kind of just a recurring villain, and I don't really feel like the Metroid series "needs" Ridley (or any character) as a primary villain anyway. I think the series functions pretty well with a unique primary threat in each game, and I'd actually be kind of disappointed if Ridley suddenly became just a villain to default to in every game since I've found some of the main villains to be pretty compelling.
Ridley was likely acting under orders from Mother Brain as in the original Metroid, as while he initiates the plot, he's still not the ultimate threat and otherwise plays the part he did in the original; a lieutenant to Mother Brain. With the exception of Gorea(because as much as I like the fight itself the lore is seriously boring), every new threat for each game has otherwise been compelling as you said. Though who knows how they'll manage Ridley chronologically after Fusion since he's dead as of Super Metroid and no other clones exist that we know of.
Speaking of chronologically and Ridley, unless they pull another clone scenario or that Ridley is part of a species that we haven't seen in full capacity yet, I'm finding it increasingly hard to justify his possible appearance in Federation Force story-wise. Don't get me wrong, I don't oppose it, but there is evidence to support it. FF is likely to take place between Zero Mission(just using it to avoid confusion since it's otherwise the original Metroid) and Return of Samus; in Tanabe's words, "like the other Primes did", and is a spinoff, so likely to take the Hunters route and cram itself as a side story between any of the three main games. This, in turn, leaves several possible scenarios:
- Prime-->Fed. Force--->Hunters
- Hunters-->Fed. Force--->Echoes
- Echoes-->Fed. Force--->Corruption
Now, if you ask me, I believe scenario 1 to be the most likely time period, which in turn would highlight the theory I have of Ridley not appearing. SOMETHING was happening with him during Echoes, most likely being reconstructed by the Pirates for Corruption, so there is reason to believe that Federation Force could establish this idea by having him not appear, maybe explaining it in a logbook or something.
By looking at the timeline, there is a two year gap between Prime and Corruption; ample time to rebuild Ridley.