Ballin's right; the framerate does drop below 60fps. As for sound, there is an explanation for that. I'm not entirely sure how the sound is processed on the N64 console itself, but I assume it's some sort of "timestretch" method. The same stuff that's used in the PCSX2 emulator. Basically, it syncs the sound up with the gamespeed, which eliminates the cracks and pops you hear when the emulator lags. The choppiness you're hearing is the sound playing in real time, but since the framerate is not perfectly 60, the audio has to stop every once in a while to let the video catch up. With timestretch, the audio is constantly refreshing to match the video speed.
This is pretty damn hardware intensive, though, especially at the degree the N64 appears to do it (under large framerate drop instances). Ironically enough, it would actually slow the game down even more

. But it's necessary if you don't want your game to sound like **** during slowdowns (which means the guys over at Nintendo obviously knew about the potential lag in some of their games due to inferior N64 hardware). All consoles employ some form of this because the real world isn't perfect and the framerate is never exactly 20 or 30 or 60 or whatever fps, and there are always dips and surges here and there. However, most of the time, games never go past + or - 5fps, so it's not too processor intensive.
I'd actually be curious to see how well modern consoles employ timestretching. It seems like most games don't push the powerful consoles' limits these days, so it'd be tough to find yourself in a situation where there is a significant fps drop.
I'll have to check myself, but it doesn't look like SSB drops below 40fps or so (on console), so the sound distortion won't be too noticeable. The audio should, however, be slowed down as well. I'm not positive on the sound itself, but with the frequency change might come that hilarious effect when you slow down videos (low, deep sounds). Again, it's not really that noticeable since the speed isn't cut by that much.
As far as I know, the sound plugins for 64 emus aren't terribly advanced (nothing like PSCX2), so while there might be some timestretching going on, it's only really enough to account for the ~5fps "tolerance" range during regular gameplay.
An option you could try is increasing (pretty sure it's not decreasing, but what the hell doI know

) the buffer size (Azimer's old driver only), which should delay the sound up to 1 second, which will decrease that choppy sound you hear during slowdowns, since it's waiting for the video to be processed before processing the audio. It looks like there is no buffer size option in the new Azimer driver (even says so in the settings!) so you're stuck with whatever "Dynamic Audio Sync" does (not even sure exactly how advanced it is or what sort of latency it uses).
TL;DR: The audio plugins for 64 emus were not designed to cope with massive framerate distortion, yet the console seems to be geared that way.