Official: SSBPD unsupported; source code released.

[Nintendude]

Another problem with looking at individual game wins is that it kind of disregards the counterpick system. Games between two players are not equivalent because the stages / characters change. Sets, on the other hand, are a lot more equivalent to each other. The only thing that changes is the number of games. IMO, more games only increases the probability that the better player will win.

2 things:

1. i wasn't even at genesis 2. i've never even been further went than south bend for FC for smash. also the only other tournament i'm listed under, several of those losses are also incorrect. that's why it's wrong.

2. the vast majority of tournaments that I won were under smurf names. I've won tournaments as mew2king, DA_RAIN, X Japan, DireVulcan, NJFoxMasta007, The_Cape77, and tons of other smashers that already exist. So yeah I don't see that working out lol

So, that's a duplicate tag problem rather than an inaccuracy problem. For the second point, certain results can manually be merged into other players as long as you tell us exactly which tournaments you entered and what you entered as.

 

[Kels88]

Yo Whoever is doing the Elo rankings should take my name out of genesis because i never went. That was my friend from Chicago, Ray Chun, who entered under my name.

 

[Kal]

Just so people know, assuming that the probability of winning does not change from match to match, the difference between a best of three and best of five is pretty much negligible until your probability of winning reaches 75% or so. For the most part, results will be identical between best of three and best of five, except in cases where the skill gap is pretty significant.

Also, I forgive you Scar. Or I apologize. I don't really know what's happening with our conversation. I just found the pedantic assessment of the degrees of pedantry ironic. Though it's pedantic for me to explain the joke.

 

[Zivilyn Bane]

Kels, got it fixed. I thought something was up there. Seemed so odd for you to lose those matches.

 

[Varist]

Has the topic of Javi come up yet?

How will data for him be obtained? Anyone have a connection to the Mexican Smash scene's TOs? Do they even keep tio files?

We only have Apex 2012 results for him and until we can dredge Javi up out of the 1500s there will be a glaring hole in our Top 10.


---


And question. If a really small local (sub 15 people) makes a formal bracket and uploads the tio file does it count towards rating? Even if the opponents are randoms?

 

[Zivilyn Bane]

Varist, I've PM'd one of the TOs for the Mexico Apex qualifier but have got no response.

And yes, locals count and are very important to building the DB. Locals play eachother and gain points and lose points like normal. Then when those players to play in larger tournaments the more skilled players gain more elo for wins over them.

 

[Pinkie Pie]

so for this tournament, which i didn't attend.
http://www.ssbpd.com/tournament/Subway Fight Night
JOS is actually Anupam
"SOS is DQ'd" is a bye.

thanks!


and i think the issue of DQ's has been resolved?

also, i flagged Vash as Kevin. when the merge happens, the Kevin from
http://www.ssbpd.com/tournament/Test Your Skills Bar Fights 6
and
http://www.ssbpd.com/tournament/Winter Game Fest VI (Melee)
and
http://www.ssbpd.com/tournament/Subway Fight Night
who beat smiles and anupam and lost to jake and oscar will be Vash.

and Kevin (the second Vash) will hereafter be named Kevin or something else

and then i accidentally flagged armada as being from texas wtf.

Way to be, Jose.

 

[LLDL]

Just noticed something. OTG didn't show up day 2 of zenith 2011, so he never played mafia or prog.

 

[Zivilyn Bane]

so for this tournament, which i didn't attend.
http://www.ssbpd.com/tournament/Subway Fight Night
JOS is actually Anupam
"SOS is DQ'd" is a bye.

thanks!


and i think the issue of DQ's has been resolved?

also, i flagged Vash as Kevin. when the merge happens, the Kevin from
http://www.ssbpd.com/tournament/Test Your Skills Bar Fights 6
and
http://www.ssbpd.com/tournament/Winter Game Fest VI (Melee)
and
http://www.ssbpd.com/tournament/Subway Fight Night
who beat smiles and anupam and lost to jake and oscar will be Vash.

and Kevin (the second Vash) will hereafter be named Kevin or something else

and then i accidentally flagged armada as being from texas wtf.

If this is correct then now we have no sets in the DB for Kevin. Can you please just list which tournaments should be under Kevin, and which should be under Vash. And possibly encourage Kevin to pick a more unique name? Even if it's just Kevin (last initial). Thank you.

 

[darkatma]

Hey can someone take a look at my rankings, near the bottom, there are a bunch of losses from Pound 4 matches.. but I never attended pound 4.

Thanks lol
-DarkAtma on the ranking system

 

[OkamiBW]

What if we have a player named Max who's different from the one in Sweden? Should I title him as Max-AZ or something?

 

[JOS.fm]

If this is correct then now we have no sets in the DB for Kevin. Can you please just list which tournaments should be under Kevin, and which should be under Vash. And possibly encourage Kevin to pick a more unique name? Even if it's just Kevin (last initial). Thank you.

Ok so you identified them as Vash-RGV and Vash-Austin. That's good enough, we can change later their name later.

So, for Subway Fight Night (10/29/2011) since they both attended, the results fell under one name.

Vash-RGV defeated Smiles, Anupam and lost to Jake13 and Mojo.

Vash-Austin defeated RedShadow, Problem0 and lost to Broly and Anupam

In addition, for that tournament as well, the match between Anupam and SOS is DQ'd should be deleted because it's actually a bye for Anupam.

 

[Zivilyn Bane]

Hey can someone take a look at my rankings, near the bottom, there are a bunch of losses from Pound 4 matches.. but I never attended pound 4.

Thanks lol
-DarkAtma on the ranking system

Any idea who would have entered as you?

 

[FoxLisk]

Ok so you identified them as Vash-RGV and Vash-Austin. That's good enough, we can change later their name later.

Pinkie Pie contacted me and asked me to give them those names.

 

[darkatma]

Any idea who would have entered as you?

No I don't remember, definitely not one of my close friends.

There is another Atma though.. but he doesn't play as often as I do.

 

[Zivilyn Bane]

No I don't remember, definitely not one of my close friends.

There is another Atma though.. but he doesn't play as often as I do.

Ok I got it fixed. I just assigned those sets to "Atma" since that's what the actual .tio file says for it.

 

[Pinkie Pie]

Sorry for the vas(h)t confusion guys. Thanks for fixing.

 

[Kal]

It surprises me that one state gets three players with the same name.

 

[KishPrime]

Kal, I sense...suspiciousness.

It's a trap!

 

[Admiral!]

Some of the tournaments are displaying multiple dates. For example Game Ever Super Smash Sunday says that it took place on 03/04/2012. But if you look at a player who attended it like VTS it says his first match took place on 03/03/2012, but all other matches took place on 03/04/2012.
This only happens for SOME tournaments. Others have no problems at all. Sorry if someone has already mentioned this before.

 

[ajajayjay]

Sorry if this has been asked before, but don't all matches count equally in an Elo rating system? regardless of WHEN they were played? As in, do Mew2king's losses from 2006 and 2007 get weighed equally with matches today?

If so, is it possible to set a 1 or 2 year range of influence on which matches affect Elo rating?

I know Elo ratings correct themselves over time, but I think its weird that Ken is so high when he's not relevant anymore

 

[FoxLisk]

Some of the tournaments are displaying multiple dates. For example Game Ever Super Smash Sunday says that it took place on 03/04/2012. But if you look at a player who attended it like VTS it says his first match took place on 03/03/2012, but all other matches took place on 03/04/2012.
This only happens for SOME tournaments. Others have no problems at all. Sorry if someone has already mentioned this before.

yeah... thats because i did something silly a while ago. i probably just failed to clean up some of the ones that got messed up. sorry.

Sorry if this has been asked before, but don't all matches count equally in an Elo rating system? regardless of WHEN they were played? As in, do Mew2king's losses from 2006 and 2007 get weighed equally with matches today?

If so, is it possible to set a 1 or 2 year range of influence on which matches affect Elo rating?

I know Elo ratings correct themselves over time, but I think its weird that Ken is so high when he's not relevant anymore

Ken is so high at least partially because tournaments were uploaded out of order. i'll re-run the ratings now.

Otherwise... yes, matches all count equally, which means of course your second paragraph cannot be.

 

[Zivilyn Bane]

The thing about it is if Ken didn't get so good and rated so high back then, the pros of today wouldn't be as high rated. It's all based off who you beat and who you lose to.

Ideally the elite players like Armada and Mango will be over 2200. They just couldn't obtain those ratings if they aren't able to beat high rated players that came before them.

 

[Fortress | Sveet]

Yeah but time matters. Beating Ken 5 years ago and beating Ken today mean totally different things, even though he has the same Elo and gives the same Elo from beating him.

 

[Zivilyn Bane]

Yeah but time matters. Beating Ken 5 years ago and beating Ken today mean totally different things, even though he has the same Elo and gives the same Elo from beating him.

Is this going to be a problem? Can you please show me some inactive players you think are going to return to tournament action? And you guys are making a huge deal as if Ken was something like 2200. Sheridan said himself that he would doubt Ken could return and place top 5. But you know what? Right now he's rated 1576. That's pretty close to Lunin. I think he'd have a chance to beat Lunin. Or Jace, or Trail. He doesn't have an Elo that would indicate he would place very highly in a national. Lets get some more data in here before we really analyze Ken some more. He's the only example being mentioned and I'm pretty sure he'd be worthy of a 1500 rating.

Anyways, I did some analysis on a couple of recent tournaments. CodeName EPIC, which was the WC qualifier for IMPULSE, the upcoming Canadian National. And Apex 2012. I chose these tournaments because they feature a lot of players with a lot of data already, but they also highlight a couple of things that could be perceived as flaws, albeit with any type of rating system. So here are the results for the top 13 at Codename Epic from 3/3/12:
1. Wobbles (1846)
2. Axe (1743)
3. Eddy Mexico (1687)
4. Fly Amanita (1722)
5. TAI (1507)
5. Westballz (1618)
7. Likewise (1313)
7. Stabbed (1501)
9. baka4moe (1433)
9. Sung666 (1457)
9. ROFL (1416)
9. Bimbo Mexico (1449)
13.Dylion (1262)
13.J666 (1304)
13.Hyprid (1368)
13.festizzio (1279)

Elo is in (). The average rating of the players in each place is this:
1-1846
2-1743
3-1687
4-1722
5-1563
7-1407
9-1439
13-1303
Which for the exception of 4th, and 7th and 9th, is in order. So why are they not all in order? Well for one, a 40 point difference in Elo doens't mean too much, so it's understood and completely acceptable that a player 40 points below another player could place higher. Hopefully nobody is coming in to this expecting every tournament to have results in order of elo. That is simply not possible. Any type of rating system can only track a players most recent results. As far as 7th and 9th goes, one of the players tied for 7th is Likewise...which I'm pretty sure is an alias and it's actually somebody else. That person started the tournament at the base rating of 1200, so he's going to bring down the average for that placement a bit, similar to Javi at Apex 2012. Anyways, if you list the players in order of Elo the only two players outside the order are Fly who was the 3rd highest player and got 4th (and his counterpart Eddy who was 4th highest rated and got 3rd) and Likewise who wasn't rated at the beginning of the tournament. Looks pretty accurate to me.

Ok now for Apex. There's a lot of players and I wanted to point out the depth of Elo so I went with top 64.
1. Armada (2115)
2. Hungrybox (1967)
3. Mango (1942)
4. Javi (1529)
5. Dr Peepee (1839)
5. Kirbykaze (1657)
7. Hax (1736)
7. Shroomed (1847)
9. Lovage (1712)
9. MacD (1655)
9. Wobbles (1846)
9. Unknown522 (1546)
13.VaNz (1580)
13.SFAT (1685)
13.Zhu (1767)
13.Tope (1707)
17.PewPewU (1616)
17.KoreanDJ (1578)
17.DaShizWiz (1646)
17.Mew2King (1828)
17.Darc (1635)
17.Silent Wolf (1690)
17.Chillin (1595)
17.S2J (1737)
25.HBK (1595)
25.Weon-x (1343)
25.Eggm (1583)
25.Jman (1753)
25.Teczero (1519)
25.Bladewise (1484)
25.I.B. (1453)
25.Westballz (1618)
33.Tang (1408)
33.Ice (1628)
33.Kage (1626)
33.Redd (1383)
33.TAI (1507)
33.Sol (1365)
33.Amsah (1811)
33.Druggedfox (1417)
33.Leffen (1665)
33.Axe (1743)
33.ChuDat (1657)
33.Plup (1341)
33.Darrell (1536)
33.StriCNYN3 (1479)
33.th0rn (1540)
33.Reno (1346)
49.Eggz (1523)
49.Mattdotzeb (1292)
49.Swiftbass (1470)
49.Vist (1364)
49.Colbol (1496)
49.Vudujin (1297)
49.PC Chris (1655)
49.Cactuar (1551)
49.Azen (1591)
49.Linguini (1582)
49.Stabbed (1501)
49.Cyrain (1491)
49.Ambix (1324)
49.HugS (1630)
49.Ravenlord (1400)
49.ZoSo (1352)

And here is average rating of the players by finish.
1-2115
2-1967
3-1942
4-1529
5-1748
7-1792
9-1690
13-1685
17-1666
25-1544
33-1528
49-1470
So with this one obviously we see 4th place at only 1529. That was Javi who has only one tournaments worth of data. This is a flaw that would be highlighted in any system. It is a flaw actually that Elo already has an answer for, which is the performance rating formula that would actually eliminate any player from starting at a base rating, and instead give them a performance rating from which to start. If this formula was used, Javi would actually be rated 1712. It would have been 1361 after pools round 1, which is the only round that would have used the formula. He was 100% vs an average field of 1161. I recommend we use a +/- scale of 200 (instead of 400) to prevent new players from beating up on noobs and getting to 1500 easily. That puts him at 1361 and the rest of it uses the same formula we use. Other than that we also see 5th and 7th out of order. And we can attribute that to only having about 50 matches rated for KirbyKaze, who is obviously much better than his Elo. In fact we only have one match rated for him within the last year. Which means with decay he would've dropped down even more than what he was. Any thoughts?

EDIT: I also wanted to share that I would be open to ratings decay if the lack thereof actually does present itself a problem. However it's not something that I would think would be necessary until the backlog of tournaments is complete. Because at this point we obviously can't tell when a players activity starts and stops, since we're missing so much data still. Like KirbyKaze, I really doubt he was inactive for a whole year prior to Apex 2012. Again though, the lack of it needs to be a problem, right? I mean, lets see if there are examples of old school pros coming out of retirement and losing to players that shouldn't gain those elo points.

 

[Fortress | Sveet]

Ziv, idk why you are listing results with Elo as some sort of proof. Any rating system will have similar numbers. More accurate systems will have more accurate numbers. If I didn't have to leave for work, I'd take a really basic system (win percentages) and compare them and you'd see a surprisingly similar analysis.

As for a list of players: PC Chris and Korean DJ both took YEARS off after being top 3 in the country and returned (surprisingly) not in the top 3. In the MW alone, Dope, Drephen, Watty, CunningKitsune, Cosmo, etc have gone long periods without entering tournaments. How about Wes who left when he was ~top 10 in the US and came back now to barely make it out of pools at a local.

As for Ken's "Elo" right now, its trash. You know why? Because the system isn't actually calculating Elo properly, and you know it. For all your talk about chronology and Elo being based on the past, you fail to acknowledge that the Elo we have right now isn't actually Elo.

 

[Juggleguy]

Are gamertags case-sensitive for the database? I'm going through some old .tio files and trying to make gamertags consistent. Not sure if I should go to the trouble of fixing uppercase/lowercase differences. Similarly, how does the database identify the right event in tio? "Melee Singles" is a vague event name that could refer to singles pools or singles bracket.

Last thing: for GSS last year, we used a hybrid format that included losers bracket round robins. It was different than any format compatible with tio, so we just updated the bracket in Photoshop. Any way it can be included in the database?

 

[Zivilyn Bane]

Sveet, from a BR member I would expect a little more respect and maturity from your posts. I didn't post this analysis as any sort of "proof." I have nothing to prove especially not to you. Hyuga actually requested some analysis on the Elo ratings compared to tournament results and the accuracy in which you can predict standings based purely off Elo. This has nothing to do with me trying to prove anything. As an added benefit, the rest of the community can get an easy grasp on how the project is going as far as accuracy goes.


PC and Korean DJ are in the 1500s and 1600s, which is not breaking the system and are actually pretty accurate of their performance at recent tournaments they've played in.

The system we use IS in fact Elo which is in fact based off chronological results, so I have no idea where you're going with this. The DB doesn't automatically process them chronologically, and it can't. But we can (and do) re run the calculations every few days to keep up the accuracy.

If you want to talk more please see me on AIM or Skype because I'd rather not argue with you on smashboards.

EDIT: Juggleguy, yes case is important when uploading results. And yes I can create GSS in tio and have it rated. I'll do that tonight for you.

 

[Cactuar]

I second Ziv here. You suck, Sveet.

 

[Dan Salvato]

Are gamertags case-sensitive for the database? I'm going through some old .tio files and trying to make gamertags consistent. Not sure if I should go to the trouble of fixing uppercase/lowercase differences.

I just checked, and it appears case-sensitive, which it shouldn't be, in my opinion. It could really mess up some stats.

 

[Juggleguy]

Alright, I did my best to make all gamertags case-consistent, and I've just uploaded the following tourneys onto the SSBPD:

Rob's House 2
SWEET I
SWEET II
SWEET III
SWEET IV
UM Triweekly 1
UM Triweekly 2
UM Triweekly 3
UM Triweekly 4
UM Triweekly 5
HomeMadeRage
Masonfest
SWEET V
SWEET VI
SWEET VII

How are pools coming along for The Big House file, Ziv? Sorry again for being so lazy

 

[Strong Badam]

lmao cactuar now youre just trolling sveet

IE: We've requested the removal of case sensitivity already, its up to FoxLisk to do it

 

[Zivilyn Bane]

Lol Juggle I rarely check my email and wasn't aware you sent them. I'll get them done probably tonight. Since we can't merge files I'll also need the brackets, or a link to them.

 

[Kal]

People need to realize that the accuracy of Elo is a function of time. Assuming all players are equally active all the time, the Elo of any given player will approach their "true skill" as time passes.

It's not terribly hard to account for this. For the most part, it's just a matter of "hiding" inactive players. If Ken were to emerge and start playing again, either:

1) He would kick ***, and his Elo would stay close to where it is
2) He would not kick ***, and his Elo would drop

There's nothing wrong with this. Trying to come up with any sort of system based on when the person played just obfuscates an otherwise simple method of calculating skill.

As for the validity of the Elo itself, would FoxLisk be willing to post the source code for this stuff? Or at least the source code for how the Elo is calculated? I'm not much of a programmer, but I would be happy to help with any mathematics-related stuff.

 

[Strong Badam]

+1 to Kal's explanation

 

[Fortress | Sveet]

Sveet, from a BR member I would expect a little more respect and maturity from your posts. I didn't post this analysis as any sort of "proof." I have nothing to prove especially not to you.

Sorry for the rude comments. If you have a problem with it, you shouldn't respond in turn though. I will do my best to keep my emotions out of my posts.

PC and Korean DJ are in the 1500s and 1600s, which is not breaking the system and are actually pretty accurate of their performance at recent tournaments they've played in.

Yes but at the instance they come back they have an elevated Elo that is not representative of their actual skill. Take win percentage as a baseline ranking system to compare with your Elo predictions. Elo will correct itself over time, but so would pure win percentages over time. One will take longer than the other to reach true accuracy. The best system is the system that can most accurately describe players' skills relative to each other. In this instance (where a player has returned after a long break from the game) Elo is inaccurate. This is my primary reason for suggesting a system that has ranking decay over time. If a system is available that is just as accurate at every other time, and more accurate in this case, we should use it.

The system we use IS in fact Elo which is in fact based off chronological results, so I have no idea where you're going with this. The DB doesn't automatically process them chronologically, and it can't. But we can (and do) re run the calculations every few days to keep up the accuracy.

Is that so? I understood there was currently no reprocessing of the results in chronological order, and instead they were just being processed as they are uploaded.

If you want to talk more please see me on AIM or Skype because I'd rather not argue with you on smashboards.

Lets talk on skype next time ur on ^^

I second Ziv here. You suck, Sveet.

Cactuar you know I look up to you, it kills me when you say mean things

People need to realize that the accuracy of Elo is a function of time. Assuming all players are equally active all the time, the Elo of any given player will approach their "true skill" as time passes.

It's not terribly hard to account for this. For the most part, it's just a matter of "hiding" inactive players. If Ken were to emerge and start playing again, either:

1) He would kick ***, and his Elo would stay close to where it is
2) He would not kick ***, and his Elo would drop

There's nothing wrong with this. Trying to come up with any sort of system based on when the person played just obfuscates an otherwise simple method of calculating skill.

As for the validity of the Elo itself, would FoxLisk be willing to post the source code for this stuff? Or at least the source code for how the Elo is calculated? I'm not much of a programmer, but I would be happy to help with any mathematics-related stuff.

I agree. TBH I don't really care what system we use; even Elo is quite accurate. I just want this project to be the best it can be, and if a better ranking system is out there I believe we should use it. In my eyes I see old players coming back quite often, or at least often enough that it can cause problems. I also see skill decay (relative to the top metagame) as a very real thing in Melee. With those two things in mind, I think a system with a decaying skill value would be beneficial.

One such system that was brought up is TrueSkill. I have no experience with it, and really barely any experience with Elo, but if it stands up to Elo in all other areas and improves in this area, I don't see why we shouldn't give it a chance.

 

[Kal]

I think getting the system implemented is more important than what ranking system we use. The ranking system can be easily changed once we've got a fully functional database.

 

[Strong Badam]

When FoxLisk reprocesses, all tournaments are ran through again in chronological order; this is why sometimes the ratings are a bit out of whack when big old MLG's are processed by Ziv but then they fix themselves.

How exactly would you implement decay in a win percentage system? Just add in losses randomly over time?

 

[Fortress | Sveet]

What do you mean strong bad? I was never suggesting that. I was simply using win percentages as a base line to compare against Elo. I'm pretty sure if Ziv made the same lists with each player's win percentage alongside the Elo that similar patterns would be seen. In 90% of the cases, the exact math behind the system is irrelevant. I am sure Elo would normalize quicker than Win Percent, and I was simply saying that in order to show how improvements can be made if the system behind the ranking is more accurate.

 

[FoxLisk]

Sveet:

I appreciate your input for being well-grounded in skepticism, but at this point I think I've said enough times that I periodically re-process the tournaments in chronological order that you should know about that. If that in fact wasn't the case, your earlier criticisms would have very much deserved, but please try to be a little more aware of the state of the system.
To reiterate: I really do appreciate your skepticism, I'm not just trying to be political; criticism is a big part of what drives improvement. I just think you, as someone who is pretty involved in this, have a certain responsibility to be alert enough to notice big things like this.

To others:

Decays in rating are absolutely not happening. The short explanation of this is that implementing any sort of decay mechanic would be basically making up data where we don't have it, which is foolish. If anyone needs me to shout about data and inference and **** at them to understand this, holler and I'll do my best.

I agree that case sensitivity is unfortunate. However, Tio allows two players to enter as the same tag but with different capitalization and even play against each other, and as this project is founded to a great degree upon processing Tio generated data, I can't really get around this myself. If enough of you believe this and yell at Neal about it, perhaps this could be changed in his system, but I simply cannot accomodate Tio files, in general, without having case-sensitivity in player tags.