They chose echoes he already wanted to add. Legitimately, if they wanted to add characters important to Nintendo, Octoling was a clear inclusion.
As for your others.
Diddy's tail attacks were specifically removed, weren't they?
Shadow plays close to identically like Sonic in almost every single game he is in. It's untrue to the character to force a fully unique moveset upon him. The two signature things he does are Chaos Control and Chaos Spear- and both are animations that could be used from other characters. Considering Sonic is reduced to his primal moves, it would either be Shadow doing the same, or having two characters that don't feel like what they feel to be in game.
Chrom. Not even gonna comment. Fire Emblem is always the exception, the special case, so there is no reason for it not to be this 5th or 6th or 7th time.
Dr Mario was added for the exact reasons of Lucina and Dark Pit, in the same game. Sure, he existed before, but that isn't why he was brought back. The three of them were part of the same categorization in Sm4sh. It's just marketing that he didn't keep that moniker.
-Yes Octoling should be in, but I imagine they came too late in development to be included in any shape or form.
-Idk about Diddy, but even then I think you could make the argument that double-dipping on DK, a spin-off series would be a tad bit too much (I know about FE, but I can't see someone as Dixie or K. Rool having a CP slot full of spirits and music). I still find her omission to be very glaring though.
-Shadow can be both (either unique or semi-clone) and yet he'd be still done justice, that's a fact. Him having a unique moveset wouldn't be ''forced'' I imagine. but rather what Sakurai envisions him as. It really boils down to what the man himself and SEGA are ok with.
-Don't really like to talk about ''FE bias'', quotas and such, but the fan demand was very present in Sm4sh to the point he became a meme, I'll just leave it at there.
-Yes, the reasons and functions could've been the same, but if the dev team (that knows better than us) didn't think of him as an echo, then he most likely shouldn't be.