Gen 4 was my second favorite out of all the generations, behind Gen 3. Sure, Gen 4 had its problems, like barely any Fire types, but it was still very fun.
As far as difficult gym leaders go, Drayden, Whitney, Lenora, Byron, and Burgh were the only ones that I can remember having strong difficulties with. Drayden had a tough team all around. By the time I fought Whitney, all I had was a Bayleef and a Zubat, so getting around a Miltank with rollout wasn't that easy. Lenora's Watchog with Hypnosis and Retaliate was annoying, but wan't really that bad. Byron had a lot of strong wall that were hard to break through. The only reason why Burgh was tough was in B2W2 when I went with an all-Dark team, and only had a Liepard to my name.