Since you listed those characters, a pattern emerges. One-off characters get in if they are in the newer games of their respective franchises by the time a Smash game comes out.
Ness was in EarthBound (1994) and it was the newest game in the Mother series in 1999.
Sheik was in Ocarina of Time (1998) and it was one of two newest games in the Zelda series by Nintendo EAD in 2001.
Shulk was in Xenoblade Chronicles (2010) and it was the only game in the Xenoblade series in 2012 when Sakurai decided on the roster.
Robin and Lucina were in Fire Emblem Awakening (2012) and it was the newest game in the Fire Emblem series in 2012 when Sakurai decided on the roster.
Greninja was in Pokemon X & Y (2013) and it was going to be the newest game in the Pokemon series since Sakurai already had concept art of Greninja as early as 2012. One more thing, Greninja is not an one-off character. Pokemon are never one-off characters since they will just be available in future Pokemon games anyway.
Not to mention that Ness was the protagonist of the only Mother game that was sold internationally (which is still one of the most well-selling RPGs).
Sheik was PLANNED to be in Twilight Princess, and later was dropped. The remnants of her concept art is what was used for her Brawl design. And, if anything, Hyrule Warriors brought her back from being a one-off, even if it is a spin-off.
Shulk is a case very similar to Ness where he may be a one-off protagonist, but he is one in a VERY high-demand, high-selling, and outstanding RPG game that really shines as its own series for Nintendo.
Robin ACTUALLY represents the My Unit concept, first used back in FE7 (the first one sold internationally) and first made playable in FE12. Robin is just the first time that a My Unit has had a proper default look and a fairly set class type. Lucina.... basically coasted in off of being "Marf." In fact, barring Lucina, the
current FE representatives are the only characters to be playable and plot-tied in multiple games (Marth has 1 & 3, then their remakes 11 & 12, Ike had 9 & 10, and MyUnits count for 12 and 13). I use this specific phrasing because it excludes Eliwood and Hector who were not playable
and plot important at the same time in FE6.