Mostly the story, and how the game handled romance and child characters, I feel. In its pre-release period, the marketing for Fates specifically hyped up its story, making a great deal about how your choices matter, and how all paths are equal and whatnot. And then, when the game came out. . . the story couldn't even come
close to living up to the hype that had been made around it. The characters from the side you didn't pick behaved downright stupid, reasons for why battles happened were contrived as all hell (the actual, canonical reason why the nations of Hoshido aren't in an open war when the game begins is because the ruler of Hoshido just has the power to create
a magical peace barrier around the country that makes everyone that enters it lose the will to fight), the worldbuilding was close to non-existant (to this day, we don't even know the
name of the continent that everything in the game takes place on), the antagonists were over the top and unbelievable, and the writing was heavily favouring Hoshido when it came to portraying it in a positive light. Add to that the fact that the protagonist, Corrin, doesn't really feel like they have much in the way of agency and just get forced from point A to point B, and you end up with a grand pile of disappointment. Similarly, the secondary protagonist, Azura, just tends to withhold vital information for no reason, and ends up dying for rather contrived reasons on two out of three routes.
As for romance and child characters - well, one of the biggest selling points of the game character-wise was that on each side, you have four royal siblings, one older brother, one older sister, one younger brother, one younger sister. But since having your avatar romance characters was such a big thing in Awakening, they just had to bring it back in Fates, and so all of these siblings are adoptive siblings, and Corrin can marry them (which is where the accusations of incest and pedophilia - the younger sisters look like they're 14 at best - that another user here was throwing around come from, which I find understandable, but still rather crass). Resulting from the marriage is the child system, which is also extremely contentious for this game. Basically, in Awakening, the child characters were rather well justified for appearing, had a strong tie to the story, and (in my eyes at least) more of a reason to fight the final battle than their parents. Fates just copied the concept of child characters without any of these things - the children make no sense in the story, have no reason to be around, and their presence makes their parents look highly questionable. Essentially, they're being born and raised in pocket dimensions where time flows much faster than it does in the main one, with these pocket dimensions playing absolutely no role in the main story.
Characters in general are meanwhile perceived to be heavily gimmick-driven and lacking in deeper characterization (though I myself feel that that is a mostly unfair assessment).
And in addition to all of this, there's just the pricing of Fates. One path alone costs you as much as any full-price 3DS game. As for the other two paths? Those are something you have to get as paid DLC, which both together essentially double the cost of the game.
But as for the actual gameplay of Fates, that's actually pretty good. Conquest in particular has some really creative and challenging maps, which, combined with the various weapons, classes and skills you can pick from and play around with, gives you tons of options of how you can go about things. Birthright is far weaker in its map design and challenge, since it was created to be the "easier path", while Revelations. . . has at least creative maps, though they tend to lean a bit too much into the gimmick side of things. Nonetheless, these two paths can still be rather enjoyable, even if some of Birthright's maps can end up being a bit repetitive and simplistic, and Revelation has poor unit balance.
Of course, I can't really talk about anyone else's feelings on the matter, but these points are what seem to me the main reasons why people dislike Fates. As for me personally, I still love the
gameplay of the game, and think that Corrin has an interesting enough set of powers to remain in Smash - though I wish there was more music from Fates in Smash, other than just three or four versions of "Lost in Though All Alone". Fates has some
good music.
As for me personally, I don't think that Fates is the worst Fire Emblem game I played. That "honour" still goes to Binding Blade, aka Roy's game. **** that game, and that character. Lilina should've been the Lord of that game, and you can't convince me otherwise.
This got a bit long, but I hope that it nonetheless sums things up nicely enough.