Well, it was the polls that determined that Sonic would get in Brawl, so...
Lucas, as well as Snake, have been included not for popularity or importance to Nintendo. Lucas was included as a tribute to the Mother series which ended with Mother 3. Snake was included because his creator, Hideo Kojima, is friends with Sakurai - and he had at least one exclusive game on the NES which is as much as the Ice Climbers have. Sonic was included due to popularity and high demand, though, and had almost all of his recent games on Nintendo systems.
Naruto is an anime character that has nothing to do with Nintendo. Master Chief is a character that has nothing to do with Nintendo.
As for the names you listed: Cackletta is not nearly as important to the Mario (or Paper Mario) franchise as Ridley is to the Metroid franchise. The same goes for Bowser Jr. and Giovanni. I kinda think Mewtwo would rather be the villain for Pokemon over Giovanni or Team Rocket. If you would include a second Pokemon Trainer type-character, the originality of this one character would not be original anymore.
What I want to say is that wanting Ridley being included is not out there. I think it's pretty dumb to say "RIEDLY WLIL NEVR B IN SMASH SOTP WNATIGN HIEM", though. We had many arguments in this and other threads for why Ridley should be included, and almost all contra-arguments have been debunked. And those which weren't were based on opinion.
Ultimately, it's Sakurai's decision who is in Smash and who isn't, but speculating is what we do here, and we enjoy speculating and discussing possibilities. No need to come in and essentially say "This thread sucks, you suck, go die in a fire you *******s"