I'll take a stab at it.
First, we need to look at where the series is.
Here is sales analysis by VGChartz. The numbers I find at least match this. This is a good look at the sales trend and we'll start here. Each game has sold more than the last with Brawl selling 11.49 million units
(see here). Based on this trend, we'd expect the next game's sales to be higher. The X factor here is that there are two games, not one. This means someone who would buy the game could buy either version or both versions. This means sales would be split between both version though Nintendo may benefit as some people will buy both games instead of one. Either way, the sales will look different and it's likely the games individually will sell less because sales are split. If the 3DS and Wii U version both sell 5 million, they still sell and aggregate of 10 million. Now, there are two large forces at play that will effect sales of the game.
The Economy
The economy is anything but healthy.
Unemployment is 7.6 percent percent as of Jun 2013 and remains high despite a positive trend.
Unemployment could be much higher and there is a thought that the
government may be manipulating the ratio. Still, the number is higher than March 2008 where it was 5.1 percent. While it may not seem like a large difference, remember that the US consist of over 300 million people so a percent can be about 3 million people (give or take as the ratios don't take into account everyone).
Consumer spending has also seen a decline. What is keeping everything afloat is the Federal Reserve printing 85 billion dollars a month (over a trillion in a year). The printing is very aggressive though it helped the economy in the short term as we are seeing some increase in consumer spending and the stock market going up. However, the future is uncertain. With the Federal Reserve printing, it drives up the price of food and gas which puts pressure on consumers. Also, there is a high risk of hyper inflation
(here's Ducktales to make it simple) which can ruining the economy or the country falling back into a recession is the plan doesn't pan out. The Fed has committed to the printing until 2015. The new Smash Brothers game comes out in 2014.
The Wii U
First, each console cycle is a clean slate. So the Wii U will not be able to ride on the wave of the Wii. We are already seeing this as Wii U sales have been very weak. Currently,
Wii U sales are tracking below the GameCube (which was one of Nintendo's worst selling consoles). Also, there is a loss of consumer confidence in Nintendo and the weak line up for the Wii U will likely not help and will spread to the Wii U's sales. The Wii U saw a oost in Japan with Pikmin 3, but the Wii U will likely not see a long term increase in sales. One issues that Nintendo's 3DS is poised to cannibalize the Wii U. Nintendo wasn't smart and the 3DS and Wii U games almost mirror each other. Games like Donkey Kong Country Returns and Mario 3D are on both systems. The Smash Brothers game also shares the same name. So for consumers, there is little difference in games between the two systems and one is cheaper.
Here is an article that explains it all. Overall, the Wii U is a flawed system and it's market performance shows this.
The issue is a lack of consumer confidence and a weakened economy. The problem, as I mentioned above, is that there are two games, not one. This means consumers can get the game with either the Wii U and 3DS. So, they can pick and chose which means one can cannibalize the other. There is also critical mass. See, if more of the people you know have one, you're more likely to have that one as well.
So here is my guess: The 3DS game will likely cannibalize the Wii U version. There is little interest in the Wii U. The 3DS is cheaper and has a larger library and developer support. Parents will buy the 3DS over the Wii U and the game like up reinforces this purchase. There is also a weak economy which will prevent both from selling well. It will be one or the other. Both games will sell more like Melee. It will be a slow burn and we wont know final numbers until 3-4 years out. The 3DS one will likely sell about 6-8 million. The Wii U one may make it to 3 million. It may due better because of the fact that players will prefer the couch gameplay, but Smash alone can not justify the Wii U purchase and the lineup is weak as it. Because of this, more people will opt for the 3DS version.