Obscure characters are ok, but only once the requested characters are in. As the roster gets bigger, the demand for icons gets bigger. The more obscure characters get in over icons, the more the game moves away from 'pit your favourite Nintendo icons against each other' and more towards just marketing and gimmicks.
Even amongst casuals, the outcry for icons is higher than that for gimmicks. Again, people are complaining that there's no K.Rool, but no one would complain about no WFT if she wasn't in the game. Not putting in a gimmick character doesn't disappoint people the same way that not putting in an icon does because they don't have massive fan-bases to disappoint.
I get that they can't please everyone, but they're also not actively trying to please as many people as possible. It just seems like Sakurai cares more about using the game as a creative outlet for himself and the satisfaction of surprising people rather than pleasing the fans long-term by giving them what they ask for, because it requires less creativity on his part.
How are the characters added now gimmicks. Looking at just the Nintendo characters, who is the gimmick? Let's look at them, shall we.
Villager - Star of Animal Crossing, one of Nintendo's most successful franchises over the last decade
Wii Fit Trailer - Maybe a gimmick, but Wii Fit was incredibly successful. The main game sold over 40 million copies
Rosalina - Newer character, but she's been fairly popular and has appeared in quite a few Mairo titles since her debut
Little Mac - Was heavily requested
Greninja - Pokemon typically gets a new rep. Greninja was always a well liked Pokemon, so it made sense
Palutena - Fairly requested and the revival of Kid Icarus helped too
Robin - An oddity, but was choses due to Fire Emblem Awakening's success and the fact he was more unique than Chrom
Shulk - Fairly requested. Xenoblade was a cult favorite and is reflected in the high prices of US copies
Bowser Jr - Requested since Brawl
Duck Hunt - Was always liked, but was more of a concept. Everyone I've heard likes this character and he's well known in the mainstream
Lucina and Dark Pit - Not counting because of the costume thing. Will note that Lucina was popular. Dark Pit would have gone over better if he wasn't a carbon copy.
So I'm not seeing who is the gimmick her. WFT, maybe. Robin? I guess. I feel your comment is more "X character I want didn't get in," but Smash 4 added a ton of well liked characters and well known ones. They were not all ones the community liked, but they made for a very strong roster nonetheless. I can agree that DK has gotten the short end of the stick, but beyond that, I don't see an issue with the characters added, and they are certainly not gimmicky.