Your render settings are most likely very different from the footage you put in. Think of rendering as an input/output kinda thing. You recorded your footage in a certain way and then inputted it to Vegas, but when you rendered it, the output was completely different from your original Melee files that you put in, and that made it look awful. You should try matching your render settings to your Melee footage, then hitting render.
Sooo for example, if your Melee footage is 720x480, you should set your render settings the same way. If you recorded using square pixels, then you should render it the same way, so on and so forth. As a personal preference, I like to render to WMV (Windows Media Video). So yeah, Render As a WMV, then try to customize the settings so that they match the original footage.
If you don't know how to check your original video settings, here's what you gotta do.
-Look for the file on your computer
-Right Click -> Properties
-Go to Details
Alternatively, you can:
-Look at the Project Media/Explorer tab in Sony Vegas.
-Find the File
-Right-Click -> Properties