It's called Horse Emblem for a reason, using any other kind of unit just isn't as enjoyable or flexible.
I would say this mostly comes down to how much units with high movement ranges (i.e. mounted units) benefit from roads increasing movement ranges even further. If you give someone like Lewyn or Shanan the Leg Ring (and maybe also the Knight Bracelet), they can easily keep up. It's the one game where Thracia 776's movement growth rate would actually have been useful.
Sigurd is nigh on invincible if you keep repairing the silver sword.
He'd still be the best unit in the game even if he got the silver sword a chapter or two later - the only negative effect would be on Seliph, who wouldn't benefit from the 50-kill bonus as early.
The holy weapons are absurdly OP
Oh, it only adds 20 to every other stat, nothing too ridiculous here
(I agree that they're fun though, Ced with Forseti or Shanan with Balmung in particular)
Weapon durability doesn't really matter thanks to repairing.
It's maybe an issue for units with holy weapons, which are expensive to repair, but otherwise it's just a minor nuisance. It shouldn't be a problem in most games though, if you manage your weapons properly (the exception being Thracia 776, of course, since money basically doesn't exist there, and you have to steal your way across the peninsula).