I think it comes from the fact that GTA is quite possibly the most infamous violent video game in history that has caused multiple panics among parents and it's name immediately evokes negative connotations for those types of people, and even for a number of people who are generally OK with stuff in video games find GTA reaching the limits of it. The only reason GTA hasn't had a controversy in years is because there hasn't been a new GTA in years, but GTAV was absolutely a controversial as hell single player experience. The game has you literally participate in a graphic torture scene while also having Trevor be this incredibly violent psychopath (which is the point, but it takes it to the point of excess) and his character is used to turn sexually assaulting another guy who is straight into a punch line multiple times in the game. Yes, that's unfortunately on brand for GTAV, but even if we get past the "think of the children" types, GTA is a really ****ing problematic series on multiple levels that has a reputation. There's no empowerment to any of it and it's commentary on society that sometimes gets attributed to it as a strength is absolutely at its weakest in GTAV where any commentary was immediately undercut by its design philosophy. Let's no whitewash how awful GTA can actually be and it's often mean spirited in its commentary still (and according to workers at Rockstar, the product of an incredibly toxic and unsustainable work environment).
Now Nintendo doesn't probably look at the empowerment angle when including Bayonetta, but Bayonetta is an unknown quantity relative to many people and her connotations are more easily navigated for inclusion in a fighting game. The second Smash Bros. X GTA comes up, you've unleashed pandora's box on to Smash and the ESRB isn't going to be kind considering they won't even seemingly allow the names of M rated games to begin with that aren't problematic in the same way the literal words Grand Theft Auto are. GTA's tendency to embrace everything controversial means there's no way around everything. Like, even if you somehow decide that GTA's controversy is not a road block, what do you do about character design? They have to use real life weapons for everything and the character will be literally grounded in real life violence for attacks if you're going to properly represent GTA at all.
GTA being a bridge too far for playable content doesn't surprise me in the least bit, there's just no real way around it being so infamous. Also, the Devil's Third content did get cut, so that's not really a point in favor of GTA in Smash to begin with.