How exactly do we measure character viability via tier lists?

[Necro'lic]

Oh looky, another tier list discussion. HOW ORIGINAL!!!

This is more of a meta discussion though. I will say right now that I ain't trying to say tier lists don't matter because they do, and I already know this. This discussion is mostly on how we measure character viability via said tier lists.

My stance is that tier lists that are early in the games life, no matter who makes them, are pretty much worthless, because people have not mastered their characters or figured out how to optimally use them yet, so the best tier list early on will mostly have a bunch of uninformed opinions or judgments and thus the only functional one involves saying "well MAYBE this character is really good and this character is really bad".

Later on however, tier lists become much easier to make with more data. But what data do we base it off of? With a history of watching Dota competitively for over 7 years now, I was always under the impression that the best metric to go off of was wins and losses in tournaments. Pick rate was the second best, but sometimes didn't really matter because a hero in Dota could be picked many times and win no games and people would not say that said hero was good because they got picked a bunch. In fact, it usually proves the opposite.

However, with this long discussion I had with multiple people on the smashbros subreddit, apparently smash tier lists are not measured just on tournament wins and losses, but on "moveset viability", representation, and difficulty playing the character. I think going by these metrics are either redundant or unwise.

I guess I will go through each of them that I've heard...

Moveset viability:

This is a redundant metric to go off of, because if a fighter's moveset is viable, that necessarily will result in tournament wins at high level, while the opposite will result in losses. Since tournament wins and losses are a much more concrete and objective measurement, it is safe to say this particular metric is not needed, since it will manifest as tournament results anyway.

Difficulty:

This usually means how difficult the fighter is to play at a baseline high-level rate. Examples would be Ryu, Icies, Rosalina, and perhaps Pac-Man. These are all very high skill floor characters that require you knowing and executing and juggling through many things in their skillsets compared to lower skill floor characters like Fox, Ike, Chrom, and Lucina. I don't think we should go by this metric either, because it gives credence to a fighter simply being easy to succeed with as them being better characters, regardless of the degree of success you will actually get with them. Imagine a low skill floor character that is already learned because they were in the last game, like Lucina for example. If she ended up doing poorly in terms of results later on as people learn the intricacies of the game more and more, would we say her ease of use contributes to her viability to win a tournament despite evidence to the contrary? No. Thus, this is a non-factor in making a tier list.

Representation:

This is very similar to the difficulty one. I will just copy-paste the metaphor I gave on reddit here:

Imagine someone picks a character and they are the only high-level player who plays said character. This player then ends up winning every single tournament they enter, yet no one else plays the character still. They continue winning with said character even though people understand the matchup (because they have to because they know this player will be a problem if they don't).

They are the only representation for that character. Is that character not considered to be good?

Conversely, let's take a bunch of high level players playing one character who never enters top 8, but they stick with them because they like the playstyle or whatever. They develop their metagame, and get more players into the fold, yet still never make top 8. Is that character not considered bad?

I think the metaphor speaks for itself. It may not be the best one ever, but it gets my point across well enough. Even if one person is representing a fighter, if that fighter gets massive results, even when people learn the matchup, is that supposed to be on the individual skill of the player or the fighter? I would say a combination of both, but the viability of said character is based on wins and losses, so even if it wasn't that player, theoretically another player just as good should do just as well. It's because of this that I see tier lists as viability of characters independent of individual skill, and thus this ends up being a moot point.

--------------------------------------------------------------------------------

Conclusion: I think tier lists should base character viability solely on high-level play results, be it wins or losses.

Feel free to counter with any points or against some of mine, but I'd rather have the most objective tier list we can get, and tournament results are the most objective measurement we have, while the others are rather subjective in nature.

 

[tronixton]

I think smash central made a tier list set up like this, it just appears to have the flaw of some characters being underepresented. Nonetheless, its a fair point

 

[b2jammer]

Oh looky, another tier list discussion. HOW ORIGINAL!!!

This is more of a meta discussion though. I will say right now that I ain't trying to say tier lists don't matter because they do, and I already know this. This discussion is mostly on how we measure character viability via said tier lists.

My stance is that tier lists that are early in the games life, no matter who makes them, are pretty much worthless, because people have not mastered their characters or figured out how to optimally use them yet, so the best tier list early on will mostly have a bunch of uninformed opinions or judgments and thus the only functional one involves saying "well MAYBE this character is really good and this character is really bad".

Later on however, tier lists become much easier to make with more data. But what data do we base it off of? With a history of watching Dota competitively for over 7 years now, I was always under the impression that the best metric to go off of was wins and losses in tournaments. Pick rate was the second best, but sometimes didn't really matter because a hero in Dota could be picked many times and win no games and people would not say that said hero was good because they got picked a bunch. In fact, it usually proves the opposite.

However, with this long discussion I had with multiple people on the smashbros subreddit, apparently smash tier lists are not measured just on tournament wins and losses, but on "moveset viability", representation, and difficulty playing the character. I think going by these metrics are either redundant or unwise.

I guess I will go through each of them that I've heard...

Moveset viability:

This is a redundant metric to go off of, because if a fighter's moveset is viable, that necessarily will result in tournament wins at high level, while the opposite will result in losses. Since tournament wins and losses are a much more concrete and objective measurement, it is safe to say this particular metric is not needed, since it will manifest as tournament results anyway.

Difficulty:

This usually means how difficult the fighter is to play at a baseline high-level rate. Examples would be Ryu, Icies, Rosalina, and perhaps Pac-Man. These are all very high skill floor characters that require you knowing and executing and juggling through many things in their skillsets compared to lower skill floor characters like Fox, Ike, Chrom, and Lucina. I don't think we should go by this metric either, because it gives credence to a fighter simply being easy to succeed with as them being better characters, regardless of the degree of success you will actually get with them. Imagine a low skill floor character that is already learned because they were in the last game, like Lucina for example. If she ended up doing poorly in terms of results later on as people learn the intricacies of the game more and more, would we say her ease of use contributes to her viability to win a tournament despite evidence to the contrary? No. Thus, this is a non-factor in making a tier list.

Representation:

This is very similar to the difficulty one. I will just copy-paste the metaphor I gave on reddit here:



I think the metaphor speaks for itself. It may not be the best one ever, but it gets my point across well enough. Even if one person is representing a fighter, if that fighter gets massive results, even when people learn the matchup, is that supposed to be on the individual skill of the player or the fighter? I would say a combination of both, but the viability of said character is based on wins and losses, so even if it wasn't that player, theoretically another player just as good should do just as well. It's because of this that I see tier lists as viability of characters independent of individual skill, and thus this ends up being a moot point.

--------------------------------------------------------------------------------

Conclusion: I think tier lists should base character viability solely on high-level play results, be it wins or losses.

Feel free to counter with any points or against some of mine, but I'd rather have the most objective tier list we can get, and tournament results are the most objective measurement we have, while the others are rather subjective in nature.

I generally agree with the points brought up here, though I'd like to note that tournament results are largely a product of these metrics, not necessarily a replacement or alternative. Over the long term:

• Characters with better kits will beat out characters with weak or ineffective kits and will win more frequently as a result.
• Characters with simpler or more forgiving playstyles will perform more consistently and thus win more frequently than more difficult characters, even if the latter are otherwise objectively better.
• More popular characters will win more frequently simply because they are played in more games, though they will probably lose more frequently as well unless the imbalance is particularly extreme ().

My only real concern is the actual math involved. Every W-L record needs to be consolidated into a single number for the sake of consistent and reliable sorting. This could be done in multiple ways - a straight win percentage (Wins/Total Games), a win-loss ratio (Wins/Losses), some points-based system (e.g. 3 points for a win, 1 for a loss), or even something else entirely - and each of these methods would weigh wins, losses, and general representation somewhat differently.

Then there's the issue of determining the tiers themselves - how many blocks do you use, and where do you draw the lines between them? There are possibly multiple ways to create the tiers (e.g. fixed score thresholds for each tier vs. a fixed number of characters for each tier), and again each method would be biased slightly differently.

The overall concept is sound and one I can totally get behind, but you ultimately have the issue of determining which sorting and tiering algorithms are the most "objective" out of the available options, which might still be somewhat arbitrary even with careful consideration.

 

[SiO2]

I think smash central made a tier list set up like this, it just appears to have the flaw of some characters being underepresented. Nonetheless, its a fair point

SmashCentral's tier list is more consistent with what Smogon's OU uses for Pokemon tiers. Not necessarily based on overall strength, but based on usage + results. That said, it follows that good and powerful characters will be used more and place higher in tournaments than bad and weak characters.

 

[Necro'lic]

That said, it follows that good and powerful characters will be used more and place higher in tournaments than bad and weak characters.

That isn't always a guarantee, and I already explained why it isn't a good idea to care about characters being used more, because that's not a direct correlation with how likely they are to win.

To make a quick example, do more casual people online say that if they pick whatever everyone else is picking, they will win? No. They say that everyone else is picking X because they seem to be good, so I will too and win because X is good. The variable of X being played a lot doesn't factor in with how likely they are to win, it's simply an informed decision that leads to a likely conclusion. Unfortunately, there are many outliers that can come from these nevertheless informed decisions of thinking a character is good because they are played a lot. I'd rather we sidestep those issues and get straight to what actually matters.

 

[Crystanium]

I read from SmashWiki that tournament results play a factor. This isn't the only thing, but still, it's kind of faulty because not every person is going to attend tournaments using characters they don't deem viable. Jigglypuff wasn't believed to be good in SSBM. Sonic players during SSBB argued that Sonic was better than initially thought, and it turned out they were correct. Right now, I'm finding Ridley to not be a good character. No one can argue that his recovery is even decent. He's easily overcome by characters with projectiles and/or speed. I don't think he's mid-tier as some might say.

 

[b2jammer]

I read from SmashWiki that tournament results play a factor. This isn't the only thing, but still, it's kind of faulty because not every person is going to attend tournaments using characters they don't deem viable. Jigglypuff wasn't believed to be good in SSBM. Sonic players during SSBB argued that Sonic was better than initially thought, and it turned out they were correct. Right now, I'm finding Ridley to not be a good character. No one can argue that his recovery is even decent. He's easily overcome by characters with projectiles and/or speed. I don't think he's mid-tier as some might say.

That's the thing, though. Early tier lists will always be wrong to some degree: theorycrafting will miss significant nuances that might be discovered later on (), and the low sample size of tournament results might cause some less popular characters to be underestimated (). That said, while tournament results may not be immune to their own form of bias, they are the least subjective metric we have, and their bias will eventually correct itself as more matches are played and players gain a better understanding of the game as a whole.

 

[Necro'lic]

That's the thing, though. Early tier lists will always be wrong to some degree: theorycrafting will miss significant nuances that might be discovered later on (), and the low sample size of tournament results might cause some less popular characters to be underestimated (). That said, while tournament results may not be immune to their own form of bias, they are the least subjective metric we have, and their bias will eventually correct itself as more matches are played and players gain a better understanding of the game as a whole.

Precisely this. You said it better than I could.

 

[b2jammer]

For the sake of further discussion and to get the ball rolling, here is one system that's simple to understand and implement, if potentially problematic in ways I haven't anticipated.

Sorting: Win Percentage (W/[W+L])
Win percentage does not take representation frequency into account at all, so a character with a record of 10-5 and another with a record of 20-10 would be tied on the tier list at ~67%. It also avoids the W/L ratio issue of weighing high win rates much more heavily than low ones, though this may become insubstantial when the actual sorting happens.

Tiering: Three tiers with fixed score thresholds (in interval notation):

• A: [67, 100]
• B: [33, 67)
• C: [0, 33)

Characters are considered tied if their win percentages are within 1% of each other (i.e. |A-B| <= 0.01), though this specific threshold was chosen arbitrarily.

Matches are almost always played Bo3, so if a character wins 2 out of every 3 games they play (and is thus A tier), chances are that they can win a set completely by themselves. If a character is winning less than 1 match out of every 3 (and is thus C tier), they will most likely need a secondary either because they are outright unviable on their own or they just have really bad matchups against the high tiers. A character that is somewhere in the middle (a B tier winning ~1.5 of every 3 games) still has a good chance of winning a set solo, but may still need a secondary to cover a few bad matchups.

Any further granularity in the tiers is probably unnecessary; the question most often asked about character viability is the need for a secondary, and low-tier tournaments often don't require any specific cut-off point beyond "is this character normally unviable?". Direct comparisons between characters would probably be best resolved through the actual scores.

 

[Crystanium]

That's the thing, though. Early tier lists will always be wrong to some degree: theorycrafting will miss significant nuances that might be discovered later on (), and the low sample size of tournament results might cause some less popular characters to be underestimated (). That said, while tournament results may not be immune to their own form of bias, they are the least subjective metric we have, and their bias will eventually correct itself as more matches are played and players gain a better understanding of the game as a whole.

Well, for Ridley's sake, I hope you're right. Trela recently won a tournament with Ridley, but against Yoshi, it was a struggle. Big body characters tend not to succeed, mainly due to their size. Yoshi was comboing left and right, but Ridley struggled.

 

[b2jammer]

Well, for Ridley's sake, I hope you're right. Trela recently won a tournament with Ridley, but against Yoshi, it was a struggle. Big body characters tend not to succeed, mainly due to their size. Yoshi was comboing left and right, but Ridley struggled.

That's definitely true; I should have clarified that I wasn't necessarily arguing against your point on Ridley (though I do think heavies in general are going to fare much better in Ultimate than they have in the other games). I was more bouncing off your statement about how Melee Jiggs and Brawl Sonic were very underrated at the start of each game's life, and how that was more of a reflection of players generally lacking experience with the game rather than any specific tiering methodology.

 

[Crystanium]

That's definitely true; I should have clarified that I wasn't necessarily arguing against your point on Ridley (though I do think heavies in general are going to fare much better in Ultimate than they have in the other games). I was more bouncing off your statement about how Melee Jiggs and Brawl Sonic were very underrated at the start of each game's life, and how that was more of a reflection of players generally lacking experience with the game rather than any specific tiering methodology.

I think Jigglypuff is a special case though because the game had been out for years. I'm thinking as early as 2008 when Mang0, I believe, defeated M2K. SSBM came out in 2001.

 

[Necro'lic]

I think Jigglypuff is a special case though because the game had been out for years. I'm thinking as early as 2008 when Mang0, I believe, defeated M2K. SSBM came out in 2001.

It honestly proves my point that tournament results are all that mattered, and the moment Jiggs got some, her tier placement went up despite not changing her moveset at all. If tiers were based on nebulous things like moveset viability or difficulty playing them, she wouldn't have moved much if at all despite getting results.

 

[Crystanium]

It honestly proves my point that tournament results are all that mattered, and the moment Jiggs got some, her tier placement went up despite not changing her moveset at all. If tiers were based on nebulous things like moveset viability or difficulty playing them, she wouldn't have moved much if at all despite getting results.

I agree. I think really it comes down to two main points that are immutable: frame data and match-ups. One can play as the worst character and perhaps do well against some people, but then you have to ask yourself just how updated is the player in terms of match-up knowledge and experience? If I lose to Ganondorf, it could just be that the player knows how to deal with Samus better than I know how to deal with Ganondorf in spite of Samus having a better neutral, projectiles, faster frame data, and better recovery. This would just mean that I lack the knowledge and experience against Ganondorf.

 

[Necro'lic]

I agree. I think really it comes down to two main points that are immutable: frame data and match-ups. One can play as the worst character and perhaps do well against some people, but then you have to ask yourself just how updated is the player in terms of match-up knowledge and experience? If I lose to Ganondorf, it could just be that the player knows how to deal with Samus better than I know how to deal with Ganondorf in spite of Samus having a better neutral, projectiles, faster frame data, and better recovery. This would just mean that I lack the knowledge and experience against Ganondorf.

You're right, it could. But at that point, what we need is not to turn towards speculation and theorycraft, but to get more samples of data. Otherwise, we should say Ganondorf-Samus matchup is in Ganon's favor simply because he beat you specifically. Or go by individual skill, which again, tier lists should be as devoid of individual skill as possible.

 

[Crystanium]

You're right, it could. But at that point, what we need is not to turn towards speculation and theorycraft, but to get more samples of data. Otherwise, we should say Ganondorf-Samus matchup is in Ganon's favor simply because he beat you specifically. Or go by individual skill, which again, tier lists should be as devoid of individual skill as possible.

Well, I do pretty well against Ganondorf, given the traits Samus has. Ganondorf needs to approach or he could stand there and receive minor damage. Samus can avoid Ganondorf for the most part. Empirical data is useful, but so is being theoretical. Tier lists will require results, and those are found from tournaments. But again, tournament results fail to paint the big picture when many characters are dismissed.

 

[Necro'lic]

Well, I do pretty well against Ganondorf, given the traits Samus has. Ganondorf needs to approach or he could stand there and receive minor damage. Samus can avoid Ganondorf for the most part. Empirical data is useful, but so is being theoretical. Tier lists will require results, and those are found from tournaments. But again, tournament results fail to paint the big picture when many characters are dismissed.

If characters are dismissed, then they have not enough "field work" at the upper echelons to put a judgment on them. If there is a super low pick rate or none at all, you can't then go on and put some theoretical stuff into a tier list built on empirical data, else it muddies the waters too much. For example, let's say Little Mac is not played nearly enough to get a good amount of data for wins and losses. The better thing to do is to rank him by the little data you do have rather than theorize why few people play him, because the reasoning behind players not picking him because "he's just bad" has just as much credence as players not picking him because "I don't like his playstyle". There are many reasons characters are picked less than others. Another reason why basing tier lists on pick rate is a terrible idea.

 

[Crystanium]

If characters are dismissed, then they have not enough "field work" at the upper echelons to put a judgment on them. If there is a super low pick rate or none at all, you can't then go on and put some theoretical stuff into a tier list built on empirical data, else it muddies the waters too much. For example, let's say Little Mac is not played nearly enough to get a good amount of data for wins and losses. The better thing to do is to rank him by the little data you do have rather than theorize why few people play him, because the reasoning behind players not picking him because "he's just bad" has just as much credence as players not picking him because "I don't like his playstyle". There are many reasons characters are picked less than others. Another reason why basing tier lists on pick rate is a terrible idea.

I think we're in agreement on certain aspects, but regarding theoretical stuff, that still relies on empirical data. To make an analogy, science uses empirical data, but that doesn't mean the philosophy of science is simply ignored. In fact, the philosophy of science brings up issues that science has. Empirical data isn't useful if you don't interpret it with elegance, simplicity, explanatory power, &c.

 

[b2jammer]

I think we're in agreement on certain aspects, but regarding theoretical stuff, that still relies on empirical data. To make an analogy, science uses empirical data, but that doesn't mean the philosophy of science is simply ignored. In fact, the philosophy of science brings up issues that science has. Empirical data isn't useful if you don't interpret it with elegance, simplicity, explanatory power, &c.

I think I understand what you're getting at here, but there might be some confusion between the means and the ends going on (and this isn't specifically directed at you - this has been a point of debate around tier lists for a long time).

Let me put it this way: stating a character is in a specific tier placemet is like drawing a point in a scatterplot. A tier list or scatterplot is simply another, more convenient way to analyze a dataset. While it may help greatly when drawing a conclusion, such a plot or list is not the conclusion by itself. "Little Mac is low tier" is a statement that can be experimentally verified through tournament results, but doesn't really explain anything on its own.

"Little Mac is low tier because his strong ground presence does not make up for his weak air game and abysmal recovery" is a possible explanation. So is "Little Mac is low tier because he has difficulty KO'ing opponents despite his abundance of projectiles and strong kill moves," which is obviously untrue, but neither claim can be proven or disproven through results alone.

This is where the theorycrafting and philosophy you've talked about comes in, not in actually creating the tier list, but rather in explaining and justifying it. When you cleanly separate the data from more subjective aspects like game design and expert opinion, you can begin to draw larger conclusions on not just specific characters, but on broader archetypes or even the engine as a whole. Trying to do the reverse as many tier lists have done leaves them vulnerable to personal bias or groupthink, and may thus reduce their accuracy as a reflection on the metagame.

 

[Necro'lic]

I think I understand what you're getting at here, but there might be some confusion between the means and the ends going on (and this isn't specifically directed at you - this has been a point of debate around tier lists for a long time).

Let me put it this way: stating a character is in a specific tier placemet is like drawing a point in a scatterplot. A tier list or scatterplot is simply another, more convenient way to analyze a dataset. While it may help greatly when drawing a conclusion, such a plot or list is not the conclusion by itself. "Little Mac is low tier" is a statement that can be experimentally verified through tournament results, but doesn't really explain anything on its own.

"Little Mac is low tier because his strong ground presence does not make up for his weak air game and abysmal recovery" is a possible explanation. So is "Little Mac is low tier because he has difficulty KO'ing opponents despite his abundance of projectiles and strong kill moves," which is obviously untrue, but neither claim can be proven or disproven through results alone.

This is where the theorycrafting and philosophy you've talked about comes in, not in actually creating the tier list, but rather in explaining and justifying it. When you cleanly separate the data from more subjective aspects like game design and expert opinion, you can begin to draw larger conclusions on not just specific characters, but on broader archetypes or even the engine as a whole. Trying to do the reverse as many tier lists have done leaves them vulnerable to personal bias or groupthink, and may thus reduce their accuracy as a reflection on the metagame.

Seriously, you are just nailing my mindset on the head far clearer than I did. Have you been pondering these sorts of things as long as I have without saying anything?

 

[b2jammer]

Seriously, you are just nailing my mindset on the head far clearer than I did. Have you been pondering these sorts of things as long as I have without saying anything?

Not really, actually. I've never been to any regional or national tournaments, but I have a lot of indirect experience from browsing sites like this forum or SmashWiki since the Brawl days, so I have pretty good insight into how the competitive scene works in a general sense.

It also helps that during my spare time I write a lot of short, pointless essays (that I never publish) about the games I've played, which helps me wrap my mind around more abstract concepts.

 

[Necro'lic]

Someone made a tier list based on only tournament results. Granted, it's from only one month, so the sample size is puny, but it's a start. The one BIG criticism I have with this is that it's influenced by tournament placing when it shouldn't, and ends up being more popularity contest which might sort itself out. However, it might not because this is mainly based on the faulty points system they are given. Again, tournament placings shouldn't matter for two reasons.

1: If it does, then more popular characters automatically will get higher placement simply due to probability of them winning being in their favor more than low popularity characters simply due to sheer numbers. The one Ice Climber getting 1st place is not gonna do much when Palu and Marth take up the other 7 top 8 slots.

2: The more wins that are accumulated, the more likely they are going to place higher in tournaments because you need to win more than everyone else to get high placement. Thus, win rate directly correlates to higher tournament placings anyway, so counting tournament placings as a multiplier is redundant.

I say that "points" should be accumulated by win rate alone. Heck, we could go deeper and not count each set, but each singular fight (in order to not overcomplicate sets where players switch characters). I would say the equation should be "wins/total fights". Keep it simple. So for examples:

High popularity:

Wario gets 1000 matches played over the course of a month. He wins 600 matches and loses the other 400. Wario's monthly win rate is 60%

Low popularity:

Ice Climbers get 30 matches played over the course of the month. They win 18 matches and lose the other 12. Icies' win rate is also 60%

Conclusion: Wario and Icies' are in the same tier.

No player count or pick rate involved, because what matters at the end of the day is that characters win or lose.

 

[b2jammer]

I've made a proof of concept of such a tier list on Google Sheets. Right now, it only uses data from Sky's Ultimate Invitational (with the results on the Smash.gg page). With only 51 games, the sample size is vanishingly small, nowhere near enough to produce an accurate list, but I wanted to see what such a W-L based tier list would look like at a glance. I also included the stages (even though I do nothing with them) because stage legality does have a noticeable, if small, effect on character viability.

https://docs.google.com/spreadsheets/d/1Wm3Kwbq3akVpcP1pTMjppQQBLASOWzLw7gxPFJFGJAU/edit?usp=sharing

EDIT: Added some more tournaments to the results, all of which are linked in the "Tournaments" sheet. It was surprisingly difficult to find tournaments that tracked character usage, and AFAIK none were recorded in pools (which makes sense - you want characters to be compared at their best, after all).

https://docs.google.com/spreadsheets/d/1uy-jALXlYA4d_ReGz9vcO0iUR6TvuLoQJhHon6sgSu4/edit?usp=sharing

 

[Gantelope]

I've always been fuzzy on this: when people talk about tiers, are they generally referring to high-level play? When you say "tournaments", does that mean all tournaments, or just big ones?