Anyone read this article at SourceGaming yet?
http://www.sourcegaming.info/2015/08/13/the-dangers-of-online-polling/#more-1847
I don't have time to write a full and concise response right now, but the key points:
1) Author correctly points out that the community's online polling has limitations with regards to effectively representing the entire population of Smash Ballot voters. While fair, this limitation is overstated:
a.) Unlike researching cereal or political preferences (analogies of the author), the population interested or even familiar with Smash Bros is more narrow. Those interested in voting for a Smash Brothers character -- or even aware this is possible -- is narrower still. Online communities where Nintendo/SmashBrothers fans congregate are actually a great place to poll respondents because of this fact. After all, as the article itself mentions, the Smash Ballot is itself an online poll! That is, the criticisms about the community's polls being "unrepresentative" also apply to the Smash Ballot itself, even if to different degrees. Remember, the Smash Ballot is a poll that has barely even been advertised. It is not a worldwide or even a regional census so it is time to stop treating it as anything remotely similar.
b.) The "feverish" fanbase has a great impact on the ballot than the more "casual" fanbase because:
> they are more organized -- votes are probably funneled into fewer characters rather than the wider spectrum of possible "casual" choices)
> and they probably voted several times -- probably multiple times for the same character.
Even if they have not voted multiple times themselves, dedicated fans have been overtly "campaigning" for each other's vote and for the votes of close friends and family. Consequently, we can expect that the average impact of the individual from the dedicated fanbase is greater than than the individual that is less interested and "casual".
Related to part a.) above, the author seems to confuse the Smash Ballot as actually representing (or nearly representing) the population of people interested in voting for a Smash Bros character. This seems unlikely given: the non-existence of commercials for the ballot, ballot announcements restricted to social media and Nintendo Directs followed more closely by the dedicated fanbase, the limitations of online polling in general, etc.
2.) Community survey results vary, but top characters do not vary as much as the author seems to believe. Maybe someone can help me out and post a link to that google doc compilation of community polls or any other links to community polls conducted around the internet. I don't have the numbers in front of me at this moment, but the belief that each community is a special snowflake is unfounded. I think this is apparent to anyone that has been following the polls.
3.) Not a particular focus of the article, but nonetheless relevant: Critics tend to selectively apply the idea that some characters are unknown to casual audiences. That is, the brunt of these criticisms tend to fall on K. Rool, Isaac, Inkling, Bandana Waddle Dee, etc. and others who have received many votes in community polls. This criticisms, however, applies to virtually every character popularly discussed among the community. If K. Rool/Isaac are actually doing poorly despite a significant online presence, how well are characters that do not even have that? Are we to believe this hidden casual mass is voting for them, "balancing" out results? Sounds implausible.
4). Finally, there are superior criticisms of community polls that this article and most critics repeating this overstated limitation do not address. Let me do their work for them:
a) Some community polls allow users to vote more than once. This allows the results to be vulnerable to serious manipulation (manually or automatic through bots). The Smash Ballot itself allows multiple votes, but there is a maximum (7 or 8?) per IP. More accurate polls would mimic this set-up.
b.) Some community polls do not allow the user to enter their own character. They are forced to pick from a selection of perceived-to-be popular or "valid" characters. This inflates the results of all of these characters because they receive the votes of users who cannot find their "ideal" character choice among the list.
b2.) Among these polls, characters are often listed in a set order (alphabetical, by game series, or arbitrarily). Characters listed first may receive more votes than those listed toward the end (voters might not see them or be too lazy to scroll down). More accurate polls will randomize the list so every user that takes the poll will see a different order ("cancelling" out this bias overall).
c.) More fervent character fanbases, including our own, are more likely to post community surveys to each other so they are oversampled in community polls.
In the end, the best safe-guard is replication. If we see survey results continuously replicated, we can probably have confidence in those results. Since we're in the K. Rool thread, I'l finish by saying that K. Rool is one character that has consistently appeared near the top and even at the top of many community polls, several of them thousands of votes in size. If K. Rool is not selected (which is still an open question) it's probably not because he didn't get enough votes in the Smash Ballot.