From my experience, Yoshi has a really, really hard time with Pit's Nair. He can't get in on it to Nair and his Fair and Bair are too slow and short to punish it's endframes.
Add on top of that a speedy, disjointed character with a great grab and Yoshi can be outclassed.
Pit has fast attacks burdened with decent endlag, but Yoshi has a hard time punishing this endlag because he is outranged, and as long as the Pit player is playing correctly, the endlag is irrelevant.
It's never easy to beat a Pit who knows how to use Nair. You have to hope they'll screw up and side-B into your shield at high percents.
Pit is easily edgeguarded and you can easily ledge drop > DJ > Nair to frame 1 his ledge grab, but it probably won't kill until about 150%. This can be repeated several times, but Nair tends to pop Pit up high enough that he is not forced to grab the ledge over and over.
edit - forgot to mention, it's the combination of Pit's Nair and Ftilt that really put the nail in the coffin. It's so hard to approach either with any of Yoshi's move, they cover his angle of approach seamlessly. Any attempts to break the wall can be read and punished.