It's alright, I'm not exactly the best with that here either considering my own late response. ^^;
That's true, Cygames and TCPi are different companies with different ideologies so of course there's going to be differences in how they approach their business and it won't be 1:1 in that regard. However, I brought them up because like how they did GBVS to dip their toes into different markets and expand their franchise, Pokemon has done the same with their current spinoffs. Take Pokken for instance, that game was clearly made to appeal to competitive players and it was pushed for their Pokemon World Tournament game alongside TCG and the mainline Pokémon games. It didn't sell a lot (only about a million on the Wii U), but it still got a port onto the more successful Switch and has still remained a major feature in their tournaments since. Same thing with Detective Pikachu, that game could've easily fell into obscurity if they wanted since it was just an Eshop game in Japan and only the first episode released for some time, but they still pushed for a physical game later on the 3DS, have a sequel in the works for the Switch, and had a movie this year.
In both cases we see that TCPi pursued these spinoff options despite the fact that neither moved many units, with Pokken probably only selling around two and a half million between both releases. So why did they do it? Because those games appealed to a market that they wanted to have. Pokken added an extra layer to their competitive community and they continue to support that, while Detective Pikachu (IIRC) was made to portray Pikachu in a different setting and way. In both cases there was an itch that TCPi wanted to have scratched and both games did it for them.
Now that's not to say that suddenly a new Mystery Dungeon will be made, but those two games do highlight that selling to the highest mark isn't everything. TCPi does look for ways that their spinoffs can appeal to different crowds and expand on Pokemon in ways that they're interested in. Mystery Dungeon has been a subseries that many still enjoy to this day, beloved for its fleshed out characters, charming stories, and (to a lesser extent) rougelike gameplay that no other spinoff in Pokemon has fully shared. If TCPi feels that itch is worth scratching, then they'll go after it. They have shown more interest in it then any of the other older spinoffs like Stadium/Colloseum, Ranger or Conquest (RIP all three), so there is an appeal they recognize in it. Whether it may turn into us getting another game in the future, that's up in the air, but I don't think it's as impossible as you say.
I'm kind of picking on this, but at the risk of repeating myself, the comparison of Monster Hunter 4 and Super Mystery Dungeon is still disingenuous since it ignores a number of factors exclusive to both games I've already explained. Additionally, it also ignores that what one company considers a success for their game is not the same for another as it comes down to their respective expectations. To use your comparison of MH, despite being a multimillion seller in Japan, Monster Hunter Generations selling a million in the west was considered a
success for Capcom. The reason for that is that relatively, while those numbers would be bad in Japan, the franchise had never exploded in the same way in other regions at the time, so passing that bar was a mark that their franchise was getting stronger and they were pleased with that. Same thing for games like Metroid Samus Returns or Fire Emblem Echoes: SoV, those games probably only did a million themselves worldwide, but Nintendo and IS respectively were pleased with the numbers and happy that they sold as much as they did. If Pokémon had those numbers in their mainline games, those would've been bombs, but for those examples it was a success.
So saying that since SMD didn't sell as well as MH4 on its own doesn't really tell us anything about how successful it was for TCPi. They of course would've been ecstatic with 4 million for the game, but on the other end it doesn't mean 1.63 million was a disappointment for them. It's all a matter of what TCPi was looking for with that game, and unless an interview or official statement comes out and explains the situation we can't say concretely it was a failure for them.