I hate to say this but it is easier to understand it in this way... Knuckles and Shadow work very similar to Piccolo and Vegeta respectively. Both of them were meant to be Sonic's rival, just that Knuckles came first and then joined Sonic, while Shadow came way after but he doesn't like to be too much time with him.
Sega or Sonic Team.. or Lizuka *cough cough* care about Shadow precisely because of that additional content, because, if anything, they could've just left Shadow without any gameplay on Forces, but they gave him one, and doesn't matter if he is similar to Sonic, he was meant to be in that way, he is a Hedgehog. Also, everytime we return to the classic formula everyone are clones of Sonic but climbs, but flies, etc. Shadow is the easier fit for new content in the modern era, yes, but they could've just not made that content, period.
What else...
-Shadow is edgy, he doesn't work being showed too much and alongside the main characters, there are always characters like this and people love these.
-Shadow The Hedgehog 2? ... the first one was a mess in terms of reputation, and even if it were well received, the first game actually finished anything about Shadow past that they could exploit, then in Sonic Battle Shadow seemed to have overcome the past and in 06 he left his hatred.
-Amy... yes, she evolved, now she is more like a friend with a love interest instead of a crazy fan... that's an awesome character development <3
-If anything, I think that the character that has the worst part is Tails, he is a coward... I mean, he can be but, he already defeated Eggman and Chaos and sometimes he said things like he wanted to be by himself and not depend on Sonic, I mean, what the hell man? It is like he was the only one who did not developed but went backwards.
Finally, Sega or Sonic Team or Lizuka has this weird thing about being radical with their decisions, I mean, let's see:
- Shadow and 06 were bad, let's forget about platforms and side characters.
- Unleashed was cool! let's use the boost formula other three times, only Sonic without any additional character so we play safe.
- Generations was cool, let's use Classic Sonic again with no apparent reason.
Also, HONESTLY, I believe Sega doesn't even respect Sonic... Nintendo seems to care more about giving time to their franchises (except for Starfox)
But, that's what I think