Villager doesn't wreck Diddy just because he can pocket a banana. A good Diddy will bait you into using pocket and then when Villager starts the animation where he pulls his hand back in and throws his arms up you will be hit.
Villager's Lloid rocket is very easy to deal with, Diddy outspeeds Villager and can crouch under Villager's fair.
Spaced Fair beats Villager's Nair. The best thing Villager can do to Diddy is gimp him.
I could see this as even at best, even then it still feels as if Diddy has the advantage because he deals with Villager's best options quite effectively.
I did my best to explain it. Maybe the matchup isn't even and Villager could wreck Diddy but all of this is from my experience.