Ladder/rating system sponsorship poll.

Which ladder/rating system would you like to see the ssb64 backroom sponsor?

  • Nintendude's Elo system

    Votes: 5 35.7%
  • AiB's ladder

    Votes: 5 35.7%
  • Other (mention which ladder/rating system you're voting for by posting in this thread)

    Votes: 4 28.6%
  • None (there are not worthy systems at this point)

    Votes: 0 0.0%

  • Total voters
    14

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#1
Please don't vote if you haven't followed the discussion in the BR Tourney thread or done any research. If you're voting for a ladder/rating system not included please post and mention what it is.

Which ladder/ranking system do you think the backroom should sponsor, if any?

1. Nintendude's Elo system
2. AiB's automated ladder
3. Other (mention which ladder/rating system you're voting for by posting in this thread)
4. None (there are not worthy systems at this point).


@ the leaders, I got permission from boom. AA told me to tell Surri to add a poll to the other thread, but he never responded. So I'm just gonna go ahead and make this thread.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#2
To summarize exactly what I am proposing, I believe that the BR should sponsor tournaments and other rated events (small round robins of official rated matches) and use those results to generate Elo ratings. These ratings will serve as the basis for seeding for all BR events, and other "BR-approved" tournaments. This system is not intended to be a ladder, and I'd like to emphasize that ratings and rankings are not the same thing. Rather, its purpose is to objectively rate players, strengthening the online community through several means:

1. Approved BR tournaments will seed based on Elo rating rather than with bias.
2. Players of all skill levels will have something to work for and a quantified representation of their progress.
3. The BR can easily sponsor amateur events with a rating cutoff.
4. The community tier list can be replaced with "classes" of rated players, as in chess.

Now, the beta program from the other thread isn't exactly representative of the logistics of how the system will be implemented. It is simply a means of maintaining the ratings system in the meantime. To be honest, it is incredibly easy to maintain ratings when rated events come around so infrequently. The fact that the system operates locally is thus not a problem for now.

I suppose I might as well spill the beans of this "behind the scenes" stuff that I said was confidential, since those involved have already exceeded their deadlines by several days. (but don't mention this publicly please, because insider info about BR admissions is the part that is confidential) Basically, I am being inducted into the MBR to lead Melee's Elo ratings project. They have told me that they are getting me the necessary resources and programmers for me to implement what I envision, and I will serve as the project manager. The system I will be working to develop will be implemented as a plugin to Tio and include the following features:

1. Interaction with a web database of rated players. TOs simply upload their results online to update the ratings. Nothing is local.
2. ID numbers to simplify player identification. Newly rated players will have be assigned a unique ID number to be used with the ratings database. The system will feature region encoding - for instance, the first digit of all North American players will be the same. The next 2 digits of players from the USA will be the same. The next 2 digits of players from New York will be the same, etc. This allows for easy generation of regional statistics.
3. Each player will be able to have multiple ratings tied to their ID number - one rating for each Smash game. This considerably simplifies the ratings process for players who play more than one Smash game. Player profiles are a possibility. The FIDE website is a more complex model of what I am ideally going for.

So that's an overview of what I am proposing. Knowing that this system obviously won't be functional in the very near future, what's wrong with AIB? Well, to once again summarize:

1. AIB does not use Elo ratings. When the Elo project is done, Smash 64 will have to start from scratch because they do not use the same algorithm.
2. AIB is a rankings ladder. Rankings ladders have failed countless times in Smash 64's history. The Elo system is NOT a ladder and it serves a different purpose.
3. AIB relies on a system that we as a BR have no direct control over. We can't force players to play matches through the system, we don't have access to dispute reports, we can't correct errors in results, etc.
4. BR = Smashboards. AIB =/= Smashboards. Some people think this is a meaningless argument but I think that is a problem. People are forced to make accounts on another website to participate.
5. On AIB you can't ensure the integrity of matches. Anyone can go play anyone at any time. For an official ratings system, only official (ie tournament) matches are rated.

Basically, think that AIB is doing more of what has already failed, and that it will be obsoleted and irrelevant when the other system is done. If we use EloCalc then we'll have a head start on rating players.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#3
Sorry Nintendude. I waited for days. I didn't know if you were ever gonna post so I just went ahead with the thread.

AiB pros:

- Automated rating/ranking system. Ratings and rankings are updated immediately after each player inputs match results. No waiting days or weeks for a single person who maintains the rankings to input and update match results and rating/rankings.

- It's a proven ladder/rating system. We know what were getting, there's no unknowns. With nintendude's system we don't know what were going to get or when were getting it. If ever.

- It's already customized for an online smash community (brawl), where Nintendude is making his system for a melee community that doesn't even have online play availability.

- It would be very easy to use their rating system in tourneys. All players would have to do is challenge each other via the ladder chat on AiB.

- We have AiB's main programmer (nealdt) who has made it clear he's willing to work with us to customize the ladder/ranking system to best fit our community.


Those are my main points. I stress the fact that Nintendude's system is basically just an idea at this point. An idea that wasn't even intended for ssb64. I don't know about anyone else but a system that doesn't even exist yet and wasn't intended for the ssb64 community does not sound like a good fit at this point, to me.

The only example we have of Nintendude's system right now is the beta program, which isn't very impressive at all. I'd suggest everyone download it to see what it's like if you have any doubts about AiB in comparison.

No offense at all to Nintendude, he's a good guy and he's just trying to help the melee community out. But I would be dumbfounded if anyone in the ssb64 community chose to wait for a non existent idea to materialize over a proven online smash ladder/rating system. If his system ever comes out we can certainly give it a try, but until it does there's really no comparison here.
 
Joined
Mar 14, 2008
Messages
9,343
Location
Sayonara Memories
#5
I approve of this thread.

OFFICIAL LEADER APPROVAL 2/3

herp :awesome:

Anyway, I don't see why these are mutually exclusive. If the question is which should be sponsored, then whether or not it affiliated with SWF has no bearing on it. I think both can be applied relatively well. The idea of ladders/rankings/whatever Elo wants to be classified as is to promote competitive play, and if people subscribe to both, I feel that's fine. I can't apply to either given I live in the middle of freaking nowhere, and I would be overjoyed if either sort of system existed for Australia, since the current level of play is degenerate. People have to ask their opponent to play serious or it defaults to friendly jab/N-Air only matches.

Personally, I think extending our community to AiB through such a ladder is a good idea. While their current 64 discussion/community is either a subset of SWF or non-existent, I think that with a bit of prompting it can be a useful tool for expanding the community.

I think Elo sounds like a good system though. I'm up to see it trialled at the very least.

This might sound like fence-sitting, but I want to see both out there for a trial period. The AiB ladder might not have succeeded in the past, but the removal of decay and public announcement of the reactivation may change that.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#6
Ratings aren't region specific, and they also don't make assumptions about the strengths of regions. If Australia was able to coordinate and set up a series of local events that were approved by the BR to be rated, there's no reasons why Australia cannot participate. If rated Australian players ever had a chance to face players from other parts of the world, their rating would apply just like anyone else's.
 
Joined
Mar 14, 2008
Messages
9,343
Location
Sayonara Memories
#8
So, who in the backroom is approving these events?

Will we finally have an official non-Head Council office?

The region-free nature of your Elo system is one of the major draws imo, but I feel that when you take the community region lean into consideration, the AiB ladder is definitely a boon to the community's competitive values in general.

Yep, I'm saying put both out for a period at the very least.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#9
My main issue with Nintendude's system is that...

It doesn't exist. And we don't know when it will or if it will. It's basically just an idea at this point.

We can certainly give it a try if it ever materializes, but until then let's give AiB a try.

And the Australian community can use AiB just as well as the North American community can.
 

SilentSlayers

Smash Journeyman
Joined
Aug 16, 2009
Messages
328
#10
Yep, voting AiB for all the great reasons SheerMadness is saying. I also agree with his point that the ELO system doesn't even exist yet.
 
Joined
Nov 6, 2007
Messages
9,681
#12
Voted for Nintendude's system. As someone who came into this neutrally, I have to say Nintendude's post was much, much more convincing. Sheer's points against it were pretty weaksauce IMO.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#13
I respect anyone's opinion and whatever they vote for.

But it does boggle my mind how anyone can vote for a system which doesn't exist and isn't even intended for a game with online play.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#14
The bare minimum of the system actually does exist already, in the form of that beta program. It's not a good program by any means but for now we'll likely be processing rated events infrequently enough that it gets the job done.

Also, the fact that it isn't intended for online play isn't really an argument. There are several prominent Starcraft leagues that use Elo ratings, and Starcraft is easily the most successful competitive game of all time.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#15
No offense Nintendude but that beta program is pretty bad lol.

And I don't play Starcraft but I'm sure they've crafted their Elo system to fit their online community. It's a pretty huge factor.

Does Starcraft have an automated system? Or is it maintened by one single person like this beta program would be?

And like I've maintained the whole time if your idea ever materializes and looks good we can certainly give it a try. But it just seems foolish to throw away an opportunity to give AiB a try over a system that doesn't even exist yet (and we don't even know when or if it will be finished). Does that not sound really dumb to anyone else?
 
Joined
Nov 6, 2007
Messages
9,681
#18
But it does boggle my mind how anyone can vote for a system which doesn't exist and isn't even intended for a game with online play.
I don't see how either of those things are a problem. It doesn't exist, and it may not be intended for online, but Elo systems have been proven to be successful for online games such as Chess, or as Nintendude mentioned, Starcraft.

lolkoro
 

SilentSlayers

Smash Journeyman
Joined
Aug 16, 2009
Messages
328
#19
I don't see how either of those things are a problem. It doesn't exist, and it may not be intended for online, but Elo systems have been proven to be successful for online games such as Chess, or as Nintendude mentioned, Starcraft.

lolkoro
Yeah, dunno what the second thing is about, it definitely is a proven system. But his first opinion is the strong one to me.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#20
I still don't get what this whole system not existing thing is all about. Elo algorithms exist. A preliminary method of automatically updating ratings from tio files exists. Yes, the program sucks, but it can take brackets and update ratings. It's not like we will be processing a high volume of results so I don't get what the big deal is. The program is usable.

The foundation is there and will be built upon.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#21
Look, there's thousands of different Elo variations out there. But they're all based on the same premise. AiB is an Elo variation. It just has one k-value for all the matches, not multiple which is the main difference.

The biggest factor here is the automated system, which simply makes AiB so much superior at this point. I've played in a lot of Elo variation online leagues over the years. Magic and Yahoo chess back in the day being two of the main ones. Their rating systems were always Elo variations, and they still are to this day. The leagues that survived had automated systems and the ones that didn't (like this beta program) died.

What online gaming community wants to wait for one guy to manually update rankings and ratings on his own? None that I know of. Which is why this beta program is simply far inferior to AiB at this point.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#22
AIB isn't Elo. AIB uses a Microsoft algorithm called TrueSkill. Read about it here:
http://research.microsoft.com/en-us/projects/trueskill/

They use TrueSkill because they believe it to be a more adequate system for Xbox Live's matchmaking. I think it is funny that you try to use "Elo not intended for online smash" as an argument when TrueSkill isn't intended for it either.

I don't see why you are under the impression that a single guy manually updates ratings. It's not like only one person has access to the program and the ratings. Do you realize how easy it would be to use a GoogleDoc to share the rating information between the backroom members?

edit:

Here is WHY Microsoft uses TrueSkill for Xbox Live:

In particular, the ELO ranking system has been used successfully by a variety of leagues organised around two-player games, such as world football league, the US Chess Federation or the World Chess Federation, and a variety of others. In video games many of these leagues have game modes with more than two players per match. ELO is not designed to work under these circumstances. In fact, no popular skill-based ranking system is available to support these games. Many one-off ranking systems have been built and are in use for these games, but none of them is general enough to be applied to such a great variety of games.

Your arguments seem based on ignorance.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#23
I don't know what exact algorithm AiB uses, but to say it's not an Elo variation seems dumb to me. Most systems are Elo variations. TrueSkill is an Elo variation modified to address flaws in the original Elo system:

TrueSkill is a Bayesian ranking algorithm developed by Microsoft Research and used in the Xbox matchmaking system built to address some perceived flaws in the Elo rating system. It is an extension of the Glicko rating system to multiplayer games.[1][2]
It still uses a lot of the same foundations Elo was built upon. Thus it's an Elo variation.

So what, multiple people can update this beta program. How on earth is that still not inferior to an automated system that updates by itself immediately after each player inputs results? Do you really not see a huge difference here? There's no need for a 3rd party to update the system, which with your beta program there would be.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#24
The only change I can find in the Glicko system is that it introduces the concept of "rating deviation." It may be an Elo variation but it still is not Elo. In fact, I'm pretty sure almost every rating system used these days is an Elo variation. So, why not stick with the original Elo, which is the most widely respected?

I think you fail to realize that AIB isn't exactly automated either. The difference is that rather than the TOs / Organizers doing the updating, the players do. They enter the results of their match and the system does the calculations. This is in contrast to the organizers entering bracket results and having a program do the calculations. Personally, I'd rather not have the players be the ones entering their results, and as I said earlier, who's to stop them from playing unauthorized rated matches?
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#25
Glad we can both finally agree that AiB is an Elo variation, which I've been saying since the beginning (my exact words were "Elo like system" I believe). Most systems are Elo variations anyway, there aren't many that are still using the original Elo system. Almost all games/sports using the Elo system are all modified and variations of the original Elo system.

You're right it's not 100% automated. The AiB website can't tell what's going on in your pj64 ssb64 match and automatically update when the match is done. It requires input from players. But compared to your beta program it's certainly far more automated. No third party is needed to update.

What do you mean unauthorized matches? The only unauthorized matches are when you've already played someone in the past 24 hours and system won't allow you to re challenge. What is there to regulate beyond re challenges (which AiB already does)??

The only possible upside I could see your beta program is changing the k-value of matches.

EDIT: And we can probably get nealdt to give the option to change k-values anyway.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#26
It's still not the same system though, and that's what matters. It would be like saying that the Playstation Move is a Wiimote variation. Of course everything new is derivative from certain benchmark developments. The only way SSB64 can ever fit into the rest of the Smash rating system is if they all use the same algorithm, and Melee will not be using TrueSkill since multiplayer matches are not a factor.

Stop brining up this k-factor thing as if it actually holds any bearing the in debate. There is no option to change k-factors with TrueSkill. Varying k-factors are built into the TrueSkill algorithm and change based on rating uncertainty (another parameter calculated by the algorithm), as opposed to Elo where k-factor is arbitrarily set and is based on a player's rating.

I don't know who this magical third party is that runs the updates. There are 2 parties: the TOs and the committee that updates ratings. TOs submit their tio files to the committee and they verify the integrity of the tournament (make sure it fits BR standards) and then update the ratings.

By "unauthorized matches" I mean any match that isn't part of a tournament. You know how people complained that ratings inflation can occur through match selectivity (like only playing people worse than you)? Well, on AIB you can't control that. That's why with any kind of REGULATED rating system, you only count the results of tournaments and other competitive events. If you have a committee process ratings directly, you can ensure that that is the case. With AIB you can't because anyone can log in at any time and play a "rated" match.

In programming there's something we refer to as the "black box." A black box is a function or set of code where you have no control over its inner workings, nor do you necessarily know how its algorithm functions; you simply give it an input and it spits out a result. It is good practice to avoid the use of black boxes as much as possible, as you want direct control over everything your program is doing. I'd argue that this situation is very similar, and that trusting the black box of AIB is not wise.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#27
I'm not going to pretend to know a lot about TrueSkill or if AiB even uses it. But I seriously find it doubtful that there wouldn't be a way for nealdt to change the value of matches played.

Not sure what you don't understand about this third party. I'm talking about your beta program, not this tio idea which has yet to even exist. I'm not going to waste time arguing over ideas which don't even exist. With your beta program you've got the players who would play their match then have to report to smashboards. Then a third party would have to go into the beta program and update the ratings. That's not needed with AiB since the players report the results on the site themselves.

Regulation over who can play who would be ******** IMO. The only person who complained about match selectivity was Surri. And we've been over why he wasn't taking advantage of the system.

We could probably argue over this subject for the next week or two, but it's pointless cause neither of us are going to change our opinions.

I will maintain that AiB is vastly superior to your beta program at this point. Until or if your tio integrated system ever materializes I doubt you're going to convince me otherwise.

And I haven't even mentioned the fact that people simply enjoy the competitiveness that a ladder brings to an online community. In your ideal scenario we wouldn't even have the option to use a ladder, just ratings by tourneys.
 

Nintendude

Smash Hero
Joined
Feb 23, 2006
Messages
5,024
Location
San Francisco
#28
Ok, I'll just raise a final point:

I am proposing a regulated / standardized ratings system to be used with tournaments and other competitive events. You are proposing a ladder in which tournament results are incorporated into the ladder. I believe that the overall premise of my idea is superior because, as we've seen in the past, ladders fail due to inactivity and people not caring anymore. A ratings system does not need frequent or continuous activity to be successful and is a more fair basis for seeding than a ladder, since ratings only take tournament matches into account while ladder rankings do not have any kind of filter on what matches can be incorporated.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#29
Ok so there are 15 backroom members and we only have 8 votes.

Where you at remaining 7 members? I'm calling you guys out. What good is a backroom if we can't even get all the members to vote on something?

Thus far:

4 votes AiB (Koro screwed up his vote)
3 votes Nintendude's Elo system
1 vote Other (Mahie votes to try both)
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#31
Yah we'll just include votes to try both in "Other". So if that's what ballin wants he needs to vote other.

And I counted everyone in the member registry thread, which Fireblaster didn't even post in.

- s2j
+ SilentSlayers

=15
 

Surri-Sama

Smash Hero
Joined
Apr 6, 2005
Messages
5,454
Location
Newfoundland, Canada!
#33
Okay, I will say this. All three of you have to play your ladder sets in the AiB chat itself. You're not allowed to play the ladder sets elsewhere. Other ladder refs/admins might think all three of you are giving free wins, and I'd rather prevent that kind of confusion from happening. So the least to say, all three of you are let off with a warning for now. But if it happens again, then consequences will definitely follow. (And btw, my source is the chatlog. The chatlogs never lie.)
This is the reply i got from the Ladder Admin from AiB when I told her me and a few other players had setup our games through msn. She had originally asked why we had such fast games.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#34
Doesn't matter Surri. Nealdt said we'd get our own ladder refs/admins so we wouldn't have to rely on brawl players who know nothing about how our community works.

So if you have questions don't pm random brawl admins about it. Ask nealdt who actually has somewhat of an understanding about whats going on.
 

firo

Smash Ace
Joined
Jul 27, 2008
Messages
600
Location
Champaign, Illinois
#35
I've been involved or observed way too many projects that have started with strong momentum but have never been carried out or have been abandoned/not supported. My vote is going to AiB because it exists and is functional. If nintendude can demonstrate results then I could be swayed.

Also, is this poll worth anything? Meaning, will the results of this poll actually dictate what the BR uses?
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#36
Also, is this poll worth anything? Meaning, will the results of this poll actually dictate what the BR uses?
That's the idea. Although I don't think anyone has a problem with giving Nintendude's thing a try if/when it does come out. So to me it's looking like we should give both a try.

It's 5-3 for AiB right now and 4 people voting for both.

Only 3 more members need to vote. Come on people.
 

KoRoBeNiKi

Smash Hero
Writing Team
Joined
Apr 26, 2007
Messages
5,950
Location
Northern NJ, 15 minutes from the GW or Manhattan
#37
This is the reply i got from the Ladder Admin from AiB when I told her me and a few other players had setup our games through msn. She had originally asked why we had such fast games.
One of the extra things on the AIB ladder is that you have to start the matches after someone accepts your request. If you put it in instantly, what that person said occurs.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#38
Who are the 3 people who haven't voted??

dandan and boom haven't even commented in this thread so assuming those 2 haven't.

But who's the third? Either way those three need to vote. This is taking way too long for such a small backroom...
 
Joined
Nov 6, 2007
Messages
9,681
#39
Well, only 2 people missing now. One of them is probably Fireblaster. Anyways, the vote is pretty split and nobody seems very against doing both, so that's probably what we're going to end up doing, unless someone has arguments against doing both I haven't heard yet.
 

SheerMadness

Smash Master
Joined
Aug 18, 2005
Messages
4,781
#40
I'm not counting Fireblaster. There's 14 registered in the member registry thread (not counting Fireblaster or s2j) and that doesn't include SilentSlayers.

So there's 15 active backroom members. Were missing one vote.

I'm guessing it's A$.

Anyway the vote right now is:

AiB - 6
Nintendude's system - 4
Other (Both) - 4

I'm fine with trying both whenever Nintendude's system actually comes out. Until then I think we need to sponsor AiB. Is anyone against that?
 
Top