Games like Space Invaders, Pac-Man, etc. were designed around difficulty simiply for profit; if a game was more difficult it would 'gobble up quarters'. Needless to say, this form of game design spilled over into the home entertainment system where designers pulled the misconception that the more difficult the game the more consumers enjoyed them.
It took game designers quite a while to break away from the archaic patterns and realize that difficulty, while still a factor, is not the prime intention of playing video games. This is actually where the primary idea behind
Kirby's Dreamland came from - Satoru Iwata and Masahiro Sakurai went about creating a game that everyone could enjoy without the difficulty being a roadblock to players.
Since a 'ridiculously difficult' game is only desired by a minority (usually the male demographic from adolescent to teens) it's much more effective to put effort into designing a game in other areas (such as visual appeal, quality of music, replay value, etc). This is where we get popular, unique and awesome games like Katamari Damashi, Pikmin, Phoenix Wright, etc.
TL;DR
Yes, 'difficulty' can be an important part of a game, but it is in the minority when compared to many other more important gameplay elements.