Retro Studios' last original game release was DKC: Tropical Freeze in February 2014. Realistically Metroid Prime 4 started sometime in 2014 as well since dev teams aren't paid to just sit around, and as happens from time to time with Nintendo projects, it probably has been in kinda dev hell for a long time since in dev it was probably turning out badly and Nintendo is really stubborn about not putting out those types of games. The fact that it was first shown with no real information in 2017 isn't terribly meaningful; it just means that that was the first point the project was successful enough Nintendo felt reasonably sure it would even launch. We saw nothing this year at E3 because it didn't fit Nintendo's E3 plans to show it there; they were pushing Smash Ultimate and didn't want to have their other reveals distract from that (it's bad marketing to compete with yourself for airspace). It's also the case that not every character who will be in the game is decided when a project plan is developed; plans do change as they go anyway. I can almost guarantee that whatever character list they had in 2015 is not exactly identical to what will launch this December. It's probably generally close but will have some deviation as things actually developed, and it very well may have even allocated resources to develop an as of the time undecided character who would be used to promote a game that hadn't come out as of the project plan being formed (which is why someone like Pyra or something crazy like a freeform labo character aren't unbelievable).
Also, things never happen for only one reason. Yes, Ridley was included because he was highly requested. Ridley was also included because he's a major component of the Metroid series and a very memorable game character in general. Ridley was also included because Metroid Prime 4 is on the horizon. Everything contributes, and given the incredibly high stakes of developing such a big title as Smash, every factor is considered and weighed. And the broader point I made wasn't "Ridley is in because of Metroid Prime 4" even. Ridley was revealed very early. There are other newcomers in Smash Ultimate too. Why choose to show off Ridley in particular so early instead of someone else? Well, again, there will be multiple reasons because nothing is ever done for only one reason, but two real big reasons might be that you want to make Metroid a focal point of your marketing or because you have several major pieces of Metroid content to reveal and want to spread them out. Of course it's all just speculation. Maybe they don't think Metroid is important at all and showed Ridley early only because the dev work on him went better than the dev work on the other newcomers so he was more ready for a demo or just because his unusually high fan demand made him fit the best with "everybody is here". Maybe Ridley didn't come with a new Metroid stage because there simply won't be one at all or because the new Metroid stage was still really buggy at the time. We can't really know anything until it is revealed, but the Metroid Prime 4 being a major focus in Smash Ultimate idea I think is plausible. We have no information that excludes its possibility at least.
EDIT: Fact check, apparently Retro isn't actually working on MP4 which raises serious questions about what they're even doing. Of course it's possible they started on the project, screwed it up, and the project got moved. We can't really know, but the underlying point was "we really don't know when MP4's development started" remains true.
If Metroid Prime 4 was in development for over two years by the time it was announced, surely they would have had something to show besides a png, at least some sort of trailer like BOTW had when it was first announced or pretty much every other game, and not still be completely silent on it over a year later.
Why choose to show off Ridley in particular so early instead of someone else?
Because Ridley has been a hugely requested character in the west since Brawl and was probably the most demanded Nintendo newcomer in the west for Smash 4, while also being a shocking inclusion because of his history with Smash and Sakurai's comments that made many people think he would never happen? E3 is for the big reveals and this is the unveiling of the game, and the only newcomer that would have topped Ridley is a huge third party character. But the order of reveal isn't important anyway, the order of reveal with the Brawl and Smash 4 newcomers pretty much had no significance, very few got revealed at the same time as a game of theirs (only one I can think of is Shulk being revealed in that Japanese direct that revealed the new 3DS Xenoblade port), with the only trend being that third parties get revealed at E3 (the only exception being Sonic, who as we know was a mid-development addition to Brawl and was probably in too early of a development state to show off at E3 2007).
I can almost guarantee that whatever character list they had in 2015 is not exactly identical to what will launch this December. It's probably generally close but will have some deviation as things actually developed, and it very well may have even allocated resources to develop an as of the time undecided character who would be used to promote a game that hadn't come out as of the project plan being formed (which is why someone like Pyra or something crazy like a freeform labo character aren't unbelievable).
The only alterations that will happen are low priority characters not making the final cut if there wasn't enough time for them and maybe a couple clones/Echoes being added. Completely new characters don't get added mid-development unless they have a case as special as Brawl Sonic's, which is pretty much not going to get replicated again, both because no other character could feasibly reach what Sonic was to Brawl, and that in the age of DLC delaying your game to add a character is completely impractical when you can just wait for DLC. For people claiming post-2015 non-Pokemon characters are going to make the base game, I wonder how many of them have actually read Sakurai's interviews and columns. Greninja's case was a special one in Smash 4, where Sakurai outright said it was an explicit exception made just for Pokemon, while also saying that if Awakening released just six months later (where Awakening released in Japan just one month before Smash 4's project plan was devised), Robin probably wouldn't have been in Smash 4. In Greninja's case he was also still decided really early on, probably before any actual development began, with Sakurai deciding him so early all he had to go off of was concept art and GameFreak's word, so it's not like he sat around waiting to see what the big hit would be.
As for your prior comment and the definitive dismissive tone it has, it reminds me a lot of when back during Smash 4 speculation of that John Knight guy who kept adamantly claiming over and over that Duck Hunt Dog would never happen and him being in Smash was nothing but a joke, and used that belief to discredit Ninka's leak even when evidence kept building towards it. Then what do you know, he ended up with a big old pile of eggs on his face. Shoddy "analysis" of the dismissed characters aside, who knows if they will make it or not, none of them are guaranteed and not every character is going to get added, but one should avoid coming in and acting sanctimonious on what characters won't make it in and which ones will, as Smash has the nasty tendency to have that blow up in people's faces, as have been demonstrated countless times; a good chunk of the internet already had that happen for Ultimate with Ridley's reveal.