I think its more that whenever they try to go for some kind of deeper plot in their games, it falls short most of the time with pretty poor fan reactions. Games like Mario Sunshine, Skyward Sword and Other M where pretty poorly received despite attempting to focus a bit more on some sort of plot.
I feel this statement is inaccurate as Skyward Sword was heavily praised, and Mario Sunshine was praised. Most of the criticism towards Sunshine was the initial knee jerk reaction (similar to Wind Waker's artstyle backlash on reveal) was that Mario was using a water nozzle and it wasn't exactly Mario 64-2 or something traditional that Mario is usually known for.
I also think the story in itself wasn't mean't to be taken too seriously, and it's storytelling is honestly in the same vain as Mario 64, just less text and more vocals in the cutscenes (most likely due to the new hardware allowing bigger file sizes) and although it had more cutscenes compared to Mario 64, there weren't really a whole lot all together. Yeah, there are those that don't want Bowser to speak or hear Bowser Jr's voice, but I feel it's hardly anything serious that effected the gameplay on any dramatic manner.
@ the topic
Now as for what I believe to be the actual reason for why they don't focus on story too much, it's because if you make a serious story with a gameplay franchise, you eventually have to end it cause if not, you'll make the story last forever and diminishes any quality when you stretch a story farther than need be. It'll end up like Mega Man X where Zero is dying and reviving every other game because they were trying to tell a proper story but they couldn't keep that story intact because it not only sacrificed gameplay, people were fans of the character and well the character had a good design and they'd have to come up with something else. Imagine if that happened with Mario and Luigi? We'd have to literally cut Luigi out of the equation if he actually died in a game, then revive him later and it'd just diminish the story logistics and sensibility.
Another great example is the Sonic Adventure series. The first two games had a great story and the second one ended on a decent note, then after that with Sonic Heroes and Shadow The Hedgehog, they completely rewrote the characters and pretty much added things that made no sense and that is because they had to continue a story that was intended to end and they ended up saying "screw it" to the logic just so they can continue a story half ***ed just cause it's there.
Mario doesn't need a story unless it's in his RPG games, that's what they are there for so I agree with
Curious Villager
in that respect. Zelda has plenty of story and lore, so does Metroid. For Pokemon, they can't develop the story too much or else face the issues I've stated previously. Now there are some spinoff Pokemon gamess that give you a story there are the Mystery Dungeon games which have a lot of personality among the Pokemon in them, there is also games like Pokemon Ranger Shadows of Almia which have an excellent story in the vain you are wanting, and the third game has a story as well.
Most importantly, if you want a "Pokemon" Pokemon game with a story, I strongly suggest playing Pokemon Colosseum and Pokemon XD: Gale of Darkness on Nintendo Gamecube as those have a story as well.
So yeah, it's for quality and to keep a franchise alive. There are games in these series out there if you look for em.