I'm a major in aerospace engineering and while I am taking propulsion next semester, I still have some opinions on the matter. The main thing that is holding us back is fuel. Rockets are incredibly fuel inefficient because they rely (as they must) on the conservation of momentum to propel them through Space. I mean, look at the shuttle; the fuel it requires to propel itself into orbit is practically the size of the shuttle itself if not more (volumetrically).
Irregardless of that, people tend to have no real perspective of how large the solar system is; it takes a long time to get to Mars...long time. And as much fun as it would be to send ships to far moons and planets...there really isn't much there that will benefit us. The best things there are to explore are the closest (relatively) to us.
Essentially, space travel will almost never be convenient and will always be costly and a hastle.