Has anyone tried something that isn’t based on ELO/trueskill for team-based games?
It feels to me like summing/averaging the ELO of a team to do matchmaking is:
a) just a hack to fit a square peg (team based matchmaking) into a round hole (individual matchmaking)
b) losing a lot of information about how intra-team dynamics would affect the result (as in player A and B play well together, better than their individual ELOs summed; but A and C play poorly together, worse than their individual ELOs summed)
I would say ELO doesn’t let you walk out of a team based game with anything other than a win or loss.
So, in sports, even if your team losses there will be players who will leave the season as All stars, or even MVPs (or vice versa, you are on a championship team but not critical at all to the team). Team esports truly is live or die by the win, and somewhat destroys the expression of players who may be league-wide top defenders/attackers/support. It’s not tracked or acknowledged.
It’s very brutal.
We need to track and factor in “advanced stats” like many sports do to some degree. You are not your ELO, but really your assists-per-game, rebounds-per-game, PPG, RBI, YPG, etc.
Match making would sort of fall into place more organically in that case possibly, since you can easily look at a team and see it needs more scoring, or more defense, versus … needs more winners or losers.
Ultimately the only thing that matters is whether your team wins or loses. This should align every player to do their best in order to win at any cost.
In the last few weeks the League of Legends Worlds Finals took place. One of the teams who made it to Round of 4 was GenG. Their midlaner is widely considered to be among the best in the world, and their team has dominated their region. Despite that, he hasn't won a single worlds. The team that won has the most legendary league player because when he shows up to worlds, he locks in and wins by any means necessary. T1 was the fourth seed in their region, and they were one game away from not even qualifying to worlds. But once they made it to worlds they locked in and won the whole thing.
I think one of the key issues with matchmaking in team based games is that they encourage solo queue instead of forming groups of reliable regulars. Having a regular group of people to play with also reduces toxicity and it lets you have clearer expectations of your allies. The power of friendship is undefeated in team based games.
I stopped playing LoL and most individualist PVP games in general when I came to the realization they ultimately peddled entertainment at the expense of someone's misery. Like you say: All that ultimately matters is whether you win, the game actually means jack fucking squat.
Once I wisened up and lost the illusion of "fun", I just haven't found any appeal in such games anymore. What kind of a human being am I if my entertainment of all things is drawn from misery? Fuck that, there's enough misery in life without fucking games making it worse.
Team and group PVP games are great because winning (or losing) is just a matter-of-fact consequence, we're there for the experience of playing together as a team and it's hell of a lot of fun for both winners and losers.
I think this is overly reductionist. Winning is fun, of course; nobody goes into a game with the intention of losing. But equating losing with human misery is a stretch. Most people playing games aren't miserable after a loss, at worst they're tilted after a tough loss or a losing streak. If losing were inherently the cause of human misery, we wouldn't be sending our kids to play organized youth sports for multiple generations. Or played chess for thousands of years.
I've been playing fighting games a lot recently, which is a pure skill-based 1v1 genre, and going up against people better than me who beat me is fun, not miserable. Obviously the ideal outcome is to win, but getting stomped by 6-time Marvel vs Capcom 2 world champion Justin Wong doesn't make me miserable, it makes me want to improve.
I'm not saying it's for everyone, I have friends who tilt off easily at every loss and yeah they shouldn't play ranked at least, but as long as your attitude is "you win or you learn" you won't be miserable.
Similarly, when I played UT'99 (by the definition it was ages ago ;-)), I loved playing against stronger players. Yeah, losing a lot, but also getting better, decreasing the gap, learning new things, and at some point being able to win against some of them (or keep the match close). Lots of them was so good - I played solely Instagib, and a lot of 1-1s - that I was happy with just making some kills. But it was a lot of fun nonetheless.
But I've also seen a lot of players that could not cope with losing - maybe overly competitive, maybe a bit immature etc. etc.
League is miserable to play, for three main reasons. The first is 30min-1hr games are the norm, and second that entire time you are playing with a group who have nearly no experience together, finally the community is genuinely horrible even on a good day. You spent an hour of your life on a team with an external locus of control who spew vitriol at everyone else and who are probably not making optimal choices themselves for nothing more than points on a scoreboard. The only way to feel accomplished in that situation is to win, because no amount of self-improvement will help someone when their teammate is trying to solo lane with Smolder, or whatever player RNG the game throws your way.
Fighting games cannot be compared to LoL in any way shape or form, they are shorter individual competitions where the only real excuse you have is internal.
> We need to track and factor in “advanced stats” like many sports do to some degree. You are not your ELO, but really your assists-per-game, rebounds-per-game, PPG, RBI, YPG, etc.
But it's besides the point, because the goal of a matchmaking ranking system (like Elo) is not to measure a player's skill but to pair player in such a way that there's a roughly even chance of either team winning.
You would assume that, but in many commercial digital multiplayer games matchmaking serves the purpode of maximizing agregate revenue. You can start from a neutral outcome objective, but then skew it in favor of likelihood of continuing to play after win/loss and increased chance of spending currency in the cash shop.
This is nudged so as to not be in your face obvious, but the skew is definetly applied.
I haven't heard of this in any major PC title, but still, it fits the general idea that Elo-like systems are there to measure relative chances of winning, not skill.
Everyone using ELO already does use something other than ELO. Game developers skew matchmaking based on:
- the amount of people in queue
- how long it's been since you last played
- total games played
- player reports
- how many cosmetics you bought (maybe)
The strength of ELO is how much it gives, for so little cost. One number representing everything. It doesn't perfectly explain the beautiful variety of nature, sure, but it's the most efficient possible abstraction. A single point value that captures 70% of what you need to know about relative skill.
Well, in every team-based PVP game one side has to lose (or a tie in sport games sometimes).
This isn't even a match-fixing because while outcome is predicted, nothing beyond matching "weak" team against "strong" team is done to ensure that outcome.
I mean, it's shitty that blizzard decides that you had enough fun crushing it and now it's someone else's get that dopamine. However, on other hand, it's nice if you're on the otherside sometimes. I will admit that sometimes it matched us against team that was so weak it wasn't even funny.
I thought about it a lot and I'm afraid there's no single ultimate solution.
It really depends on the sport and the variant you play.
Let me give you couple of examples:
(a) foosball - you have a bunch of players who play both singles and doubles. What if you have two great defenders playing together in a team loosing to much weaker, but better suited for their positions, opponents?
(b) counterstrike vs apex vs overwatch (or, to some extent dota) - what if you have a game where you can be easily carried by your team just by not sucking too much? In counterstrike one weak teammate (either lacking skill or a headset) can cost whole team a game, but in overwatch you can pick a supporting class and just chill at the back waiting for the rest of your team to win the game for you.
(c) chemistry - you probably seen that at work as well, sometimes there's a synergy between two players that only occurs in that particular setup or it can be simply a matter of playstyle.
(d) variants - in billiards you have different games that look similar but for some reason some players shine in one variant and can't get through in others.
Agreed. ELO is a crude but simple statistical model, and its primary benefit is that it's easy to understand and reason about.
I'd love for there to be a multidimensional player skill vector (aka "embedding") that encapsulates more information as well as a more nonlinear model to go on top of it.
For example, in many games, a team usually needs a support player, but a single number isn't enough to capture that for the purposes of matchmaking.
Author here: Elo is commonly used and most gamers understand it. It's nice to be able to show each player their Elo.
I love the idea to use per-player embedding and to add a nonlinear model on top. I think it may improve the model dramatically, but I'm afraid that the model may be overfitting strongly as we have only 35 games in the historical for now. Anyway - I think I'll try this out.
Yeah, 35 games is hard to work with. My rule of thumb is to not go above 1 parameter per 10 data points in order to prevent overfitting and memorization, so I'm not sure how well it will work in your case.
I used PageRank once, seemed to work well. A win was treated as a link, points would flow from the loser to the winner. The strength of links would decay over time.
"something that isn’t based on…", Many of the CPL (Cyberathlete Professional League in eSports) qualifiers were based from online rankings for players. Normally round-robin (more common known swiss) format-of tourment systems applied; go from the previous ranks, it's often hidden for different reasons, however, you look at the most recent qualifier, who performed well, then try to order everything slightly differently offline-in tournaments, then report online.
The important thing to note, when the numbers look odd-number, removing players to join-in into the event, placing the limits!
Normal orders based on name, country of origin (sometimes these individuals travel far!), time of entering tournament (sometimes people late!), then getting it to play as well as possible sometimes based on how it looks… you want some kind of catch measure, to catch 2-players early on that would play normally in the final, having 1 going home early onwards. Is this a hack? Yes and No. Losing information happens when you don't apply planning, (1-day planning then more when possible?).
someone wrote, it's Win or Lose, in these situations, sorry!
Event organisers, they like encourage the "good games" happening, when problems arise, it looks bad on every people there, the result OK, however this affects more. It is the event happening on time, people leaving at right times, everyone is OK, that people consider the "friend" aspect of tournaments, the professional points of view; this is "sportsmanship" online towards the real eSports victory.
Meanwhile the electronic groups have different "matchmaking" systems, I have even heard it is more matchmaker, it cannot be 100% perfect, however, people always try hard doing good.
Reminds me of a group of friends that played call of duty together in the CoD4/WaW days. Individually we were mid-tier players on average. As a group, we completely dominated. We all knew each other's strengths, weaknesses, and general tactics. There was often no need to communicate specifics because we knew where each other would be and what we were doing.
We were always giddy when we got matched with a clan, especially if they were a mouthy crew.
You could assign ELO at the group level. It will increase the number of possible groups by C(N, g) where N is the number of players and g is the group size. It could work if the groups are stable enough.
If you have individual ELOs, you could pre-seed the ranking using averages and then let the algorithm take over.
Excellent point about team dynamics. When trying to dynamically guess a group's Elo, it could be interesting to try and create a game-specific radar chart for each player and add those values instead.
Thanks, I didn't know what Elo meant and your comment helped me.
The Elo[a] rating system is a method for calculating the relative skill levels of players in zero-sum games such as chess or esports. It is named after its creator Arpad Elo, a Hungarian-American physics professor.
There are a bunch of GDC talks about matchmaking systems over the years too for those curious about things that have been done. For a successful game a lot of things have to be taken into account, not just skill.
Under the leaderboard tab, if the "Solution" column has an icon, it's clickable. 2nd place solution is by Jeremy Howard (of fast.ai fame), which I'd summarize as TrueSkill Through Time (Microsoft Research paper) + some overfitting on the public leaderboard (1st place was #26 in the public leaderboard).
Fun fact - our local community recently started playing billiards tournaments using a swiss system and it's really cool, but it's a tradeoff between fairness and the amount of players you can invite to a tournament. The swiss variants play at max 40 players with 6 rounds, but if you want to invite 100+ players then you have to run a 2KO ladder, otherwise the tournament would last a week.
EDIT: The best part of swiss system is it's the best value for money, because you will play throughtout the whole tournament despite your results, and while you progress the opponenets will adjust to your level, so you can have fun no matter what.
The problem is that a billiard is a big, expensive piece of equipment, and you are limited by your venue.
In chess, you only need a chessboard, which is cheap, and a table. In collectible card games like Magic: The Gathering, it is even simpler as people bring their own cards, so all you need is the table. For a LAN party, people usually bring their own computer and they have a spot for the entire event, so having everyone play at the same time is trivial.
It's also common in Magic and other card game tournaments, but slightly modified: "6 rounds of swiss, cut to top 8 the single elimination" is a common sight.
From a quick read, after a first round where you play another team at random or so, in each round players are "sorted according to their cumulative scores and [play against] opponents with the same or similar score". At the same time, it avoids having to play the same opponent twice. I don't see how one decides how many rounds to play, but not sure if that detail matters
Reading the analysis section, this is how it compares to a knock-out tournament:
> Assuming no drawn games, determining a clear winner [requires] the same number of rounds as that of a knockout tournament [...] Compared to a knockout tournament, a Swiss system has the advantage of not eliminating anyone [and] that the final ranking gives some indication of the relative strengths of all contestants, not just of the tournament winner. [However,] sometimes a player has such a great lead that by the last round they are assured of winning the tournament [such that it] does not always end with the exciting climax
Chess tournaments I've participated in had 7 rounds. The swiss system is really cool because it is a kind of sorting algorithm, where players quickly end up confronting players their level. So if you're a beginner, you still get to play to get a better ranking (instead of "whatever, I won't win anyway").
No idea what you're talking about so I looked it up
> The Shapley value is a solution concept in cooperative game theory. It was named in honor of Lloyd Shapley, who introduced it in 1951 and won the Nobel Memorial Prize in Economic Sciences for it in 2012. To each cooperative game it assigns a unique distribution (among the players) of a total surplus generated by the coalition of all players. The Shapley value is characterized by a collection of desirable properties.
This is pretty much the whole article lede on Wikipedia but I have no idea what I just read. What's a total surplus in a cooperative game here, like how much wood your team has gathered in AoE? How does it help to "assign" that, wouldn't that rather be the result of a game? Can someone dumb this down?
Has anyone tried something that isn’t based on ELO/trueskill for team-based games?
It feels to me like summing/averaging the ELO of a team to do matchmaking is:
a) just a hack to fit a square peg (team based matchmaking) into a round hole (individual matchmaking)
b) losing a lot of information about how intra-team dynamics would affect the result (as in player A and B play well together, better than their individual ELOs summed; but A and C play poorly together, worse than their individual ELOs summed)
I would say ELO doesn’t let you walk out of a team based game with anything other than a win or loss.
So, in sports, even if your team losses there will be players who will leave the season as All stars, or even MVPs (or vice versa, you are on a championship team but not critical at all to the team). Team esports truly is live or die by the win, and somewhat destroys the expression of players who may be league-wide top defenders/attackers/support. It’s not tracked or acknowledged.
It’s very brutal.
We need to track and factor in “advanced stats” like many sports do to some degree. You are not your ELO, but really your assists-per-game, rebounds-per-game, PPG, RBI, YPG, etc.
Match making would sort of fall into place more organically in that case possibly, since you can easily look at a team and see it needs more scoring, or more defense, versus … needs more winners or losers.
Ultimately the only thing that matters is whether your team wins or loses. This should align every player to do their best in order to win at any cost.
In the last few weeks the League of Legends Worlds Finals took place. One of the teams who made it to Round of 4 was GenG. Their midlaner is widely considered to be among the best in the world, and their team has dominated their region. Despite that, he hasn't won a single worlds. The team that won has the most legendary league player because when he shows up to worlds, he locks in and wins by any means necessary. T1 was the fourth seed in their region, and they were one game away from not even qualifying to worlds. But once they made it to worlds they locked in and won the whole thing.
I think one of the key issues with matchmaking in team based games is that they encourage solo queue instead of forming groups of reliable regulars. Having a regular group of people to play with also reduces toxicity and it lets you have clearer expectations of your allies. The power of friendship is undefeated in team based games.
I stopped playing LoL and most individualist PVP games in general when I came to the realization they ultimately peddled entertainment at the expense of someone's misery. Like you say: All that ultimately matters is whether you win, the game actually means jack fucking squat.
Once I wisened up and lost the illusion of "fun", I just haven't found any appeal in such games anymore. What kind of a human being am I if my entertainment of all things is drawn from misery? Fuck that, there's enough misery in life without fucking games making it worse.
Team and group PVP games are great because winning (or losing) is just a matter-of-fact consequence, we're there for the experience of playing together as a team and it's hell of a lot of fun for both winners and losers.
I think this is overly reductionist. Winning is fun, of course; nobody goes into a game with the intention of losing. But equating losing with human misery is a stretch. Most people playing games aren't miserable after a loss, at worst they're tilted after a tough loss or a losing streak. If losing were inherently the cause of human misery, we wouldn't be sending our kids to play organized youth sports for multiple generations. Or played chess for thousands of years.
I've been playing fighting games a lot recently, which is a pure skill-based 1v1 genre, and going up against people better than me who beat me is fun, not miserable. Obviously the ideal outcome is to win, but getting stomped by 6-time Marvel vs Capcom 2 world champion Justin Wong doesn't make me miserable, it makes me want to improve.
I'm not saying it's for everyone, I have friends who tilt off easily at every loss and yeah they shouldn't play ranked at least, but as long as your attitude is "you win or you learn" you won't be miserable.
Similarly, when I played UT'99 (by the definition it was ages ago ;-)), I loved playing against stronger players. Yeah, losing a lot, but also getting better, decreasing the gap, learning new things, and at some point being able to win against some of them (or keep the match close). Lots of them was so good - I played solely Instagib, and a lot of 1-1s - that I was happy with just making some kills. But it was a lot of fun nonetheless.
But I've also seen a lot of players that could not cope with losing - maybe overly competitive, maybe a bit immature etc. etc.
League is miserable to play, for three main reasons. The first is 30min-1hr games are the norm, and second that entire time you are playing with a group who have nearly no experience together, finally the community is genuinely horrible even on a good day. You spent an hour of your life on a team with an external locus of control who spew vitriol at everyone else and who are probably not making optimal choices themselves for nothing more than points on a scoreboard. The only way to feel accomplished in that situation is to win, because no amount of self-improvement will help someone when their teammate is trying to solo lane with Smolder, or whatever player RNG the game throws your way.
Fighting games cannot be compared to LoL in any way shape or form, they are shorter individual competitions where the only real excuse you have is internal.
> We need to track and factor in “advanced stats” like many sports do to some degree. You are not your ELO, but really your assists-per-game, rebounds-per-game, PPG, RBI, YPG, etc.
WAR in baseball (https://en.m.wikipedia.org/wiki/Wins_Above_Replacement) has been a pretty successful attempt at this.
But it's besides the point, because the goal of a matchmaking ranking system (like Elo) is not to measure a player's skill but to pair player in such a way that there's a roughly even chance of either team winning.
You would assume that, but in many commercial digital multiplayer games matchmaking serves the purpode of maximizing agregate revenue. You can start from a neutral outcome objective, but then skew it in favor of likelihood of continuing to play after win/loss and increased chance of spending currency in the cash shop.
This is nudged so as to not be in your face obvious, but the skew is definetly applied.
I haven't heard of this in any major PC title, but still, it fits the general idea that Elo-like systems are there to measure relative chances of winning, not skill.
CSGO at the pro level has ADR and k/d, but the stats for an igl are going to look a little different from the awper.
Heroes of newerth did something like this. They tracked gold earned, wards placed, kills, damage dealers etc etc etc.
At the beginning it was terrible. But after a year or two, it was pretty good fine tuned!
Everyone using ELO already does use something other than ELO. Game developers skew matchmaking based on:
- the amount of people in queue
- how long it's been since you last played
- total games played
- player reports
- how many cosmetics you bought (maybe)
The strength of ELO is how much it gives, for so little cost. One number representing everything. It doesn't perfectly explain the beautiful variety of nature, sure, but it's the most efficient possible abstraction. A single point value that captures 70% of what you need to know about relative skill.
And how likely it is you quit because you lost or won to many matches.
Blizzard does it. Especially in overwatch.
It's called engagement matchmaking
I'd call the one in OW1 enshittification matchmaking, because it makes games where one side has to lose.
Well, in every team-based PVP game one side has to lose (or a tie in sport games sometimes).
This isn't even a match-fixing because while outcome is predicted, nothing beyond matching "weak" team against "strong" team is done to ensure that outcome.
I mean, it's shitty that blizzard decides that you had enough fun crushing it and now it's someone else's get that dopamine. However, on other hand, it's nice if you're on the otherside sometimes. I will admit that sometimes it matched us against team that was so weak it wasn't even funny.
I thought about it a lot and I'm afraid there's no single ultimate solution. It really depends on the sport and the variant you play.
Let me give you couple of examples: (a) foosball - you have a bunch of players who play both singles and doubles. What if you have two great defenders playing together in a team loosing to much weaker, but better suited for their positions, opponents? (b) counterstrike vs apex vs overwatch (or, to some extent dota) - what if you have a game where you can be easily carried by your team just by not sucking too much? In counterstrike one weak teammate (either lacking skill or a headset) can cost whole team a game, but in overwatch you can pick a supporting class and just chill at the back waiting for the rest of your team to win the game for you. (c) chemistry - you probably seen that at work as well, sometimes there's a synergy between two players that only occurs in that particular setup or it can be simply a matter of playstyle. (d) variants - in billiards you have different games that look similar but for some reason some players shine in one variant and can't get through in others.
Agreed. ELO is a crude but simple statistical model, and its primary benefit is that it's easy to understand and reason about.
I'd love for there to be a multidimensional player skill vector (aka "embedding") that encapsulates more information as well as a more nonlinear model to go on top of it.
For example, in many games, a team usually needs a support player, but a single number isn't enough to capture that for the purposes of matchmaking.
Author here: Elo is commonly used and most gamers understand it. It's nice to be able to show each player their Elo.
I love the idea to use per-player embedding and to add a nonlinear model on top. I think it may improve the model dramatically, but I'm afraid that the model may be overfitting strongly as we have only 35 games in the historical for now. Anyway - I think I'll try this out.
Yeah, 35 games is hard to work with. My rule of thumb is to not go above 1 parameter per 10 data points in order to prevent overfitting and memorization, so I'm not sure how well it will work in your case.
I used PageRank once, seemed to work well. A win was treated as a link, points would flow from the loser to the winner. The strength of links would decay over time.
"something that isn’t based on…", Many of the CPL (Cyberathlete Professional League in eSports) qualifiers were based from online rankings for players. Normally round-robin (more common known swiss) format-of tourment systems applied; go from the previous ranks, it's often hidden for different reasons, however, you look at the most recent qualifier, who performed well, then try to order everything slightly differently offline-in tournaments, then report online.
The important thing to note, when the numbers look odd-number, removing players to join-in into the event, placing the limits!
Normal orders based on name, country of origin (sometimes these individuals travel far!), time of entering tournament (sometimes people late!), then getting it to play as well as possible sometimes based on how it looks… you want some kind of catch measure, to catch 2-players early on that would play normally in the final, having 1 going home early onwards. Is this a hack? Yes and No. Losing information happens when you don't apply planning, (1-day planning then more when possible?).
someone wrote, it's Win or Lose, in these situations, sorry!
Event organisers, they like encourage the "good games" happening, when problems arise, it looks bad on every people there, the result OK, however this affects more. It is the event happening on time, people leaving at right times, everyone is OK, that people consider the "friend" aspect of tournaments, the professional points of view; this is "sportsmanship" online towards the real eSports victory.
Meanwhile the electronic groups have different "matchmaking" systems, I have even heard it is more matchmaker, it cannot be 100% perfect, however, people always try hard doing good.
Reminds me of a group of friends that played call of duty together in the CoD4/WaW days. Individually we were mid-tier players on average. As a group, we completely dominated. We all knew each other's strengths, weaknesses, and general tactics. There was often no need to communicate specifics because we knew where each other would be and what we were doing.
We were always giddy when we got matched with a clan, especially if they were a mouthy crew.
This HN thread has some links and discussion that you might like.
https://news.ycombinator.com/item?id=26115665
https://mattmazzola.medium.com/implementing-the-elo-rating-s...
You could assign ELO at the group level. It will increase the number of possible groups by C(N, g) where N is the number of players and g is the group size. It could work if the groups are stable enough.
If you have individual ELOs, you could pre-seed the ranking using averages and then let the algorithm take over.
There’s OpenSkill https://github.com/philihp/openskill.js
Also Microsoft’s TrueSkill
Excellent point about team dynamics. When trying to dynamically guess a group's Elo, it could be interesting to try and create a game-specific radar chart for each player and add those values instead.
Sorry for pedantry but Elo is a guy's name, not an acronym.
Thanks, I didn't know what Elo meant and your comment helped me.
The Elo[a] rating system is a method for calculating the relative skill levels of players in zero-sum games such as chess or esports. It is named after its creator Arpad Elo, a Hungarian-American physics professor.
https://en.m.wikipedia.org/wiki/Elo_rating_system#:~:text=Th....
its tried and tested by multiple online games used in dota,cs2,lol,val,cod etc
it is not perfect but close enough that it didn't matter tbh, I mean multiple company do it for good reason
There are a bunch of GDC talks about matchmaking systems over the years too for those curious about things that have been done. For a successful game a lot of things have to be taken into account, not just skill.
I would guess that in team sports/esports betting some people use more advanced models, but they obviously keep it secret.
Yeah. Sports science is about evaluating individuals in a team and trying to come up with statistics that make team picking more reliable.
Moneyball is a good movie that dramatizes the dawning of this science.
Fantasy Football is a game version that's become popular for online gamblers.
-------
I've seen a paper or two floating around on the subject. But I've forgotten the names of them by now and I dunno any good references unfortunately.
Hope you find something in your search!
Kaggle had (has?) a competition for rating systems.
https://www.kaggle.com/competitions/chess/discussion/107
There are plenty of rating systems out there that perform better than Elo.
I only see the names of participants in the leaderboard, how can I access their methods?
Under the leaderboard tab, if the "Solution" column has an icon, it's clickable. 2nd place solution is by Jeremy Howard (of fast.ai fame), which I'd summarize as TrueSkill Through Time (Microsoft Research paper) + some overfitting on the public leaderboard (1st place was #26 in the public leaderboard).
For tournaments, I like the Swiss system [1], which I believe is popular in chess.
[1]: https://en.wikipedia.org/wiki/Swiss-system_tournament
Fun fact - our local community recently started playing billiards tournaments using a swiss system and it's really cool, but it's a tradeoff between fairness and the amount of players you can invite to a tournament. The swiss variants play at max 40 players with 6 rounds, but if you want to invite 100+ players then you have to run a 2KO ladder, otherwise the tournament would last a week. EDIT: The best part of swiss system is it's the best value for money, because you will play throughtout the whole tournament despite your results, and while you progress the opponenets will adjust to your level, so you can have fun no matter what.
The problem is that a billiard is a big, expensive piece of equipment, and you are limited by your venue.
In chess, you only need a chessboard, which is cheap, and a table. In collectible card games like Magic: The Gathering, it is even simpler as people bring their own cards, so all you need is the table. For a LAN party, people usually bring their own computer and they have a spot for the entire event, so having everyone play at the same time is trivial.
It's also common in Magic and other card game tournaments, but slightly modified: "6 rounds of swiss, cut to top 8 the single elimination" is a common sight.
From a quick read, after a first round where you play another team at random or so, in each round players are "sorted according to their cumulative scores and [play against] opponents with the same or similar score". At the same time, it avoids having to play the same opponent twice. I don't see how one decides how many rounds to play, but not sure if that detail matters
Reading the analysis section, this is how it compares to a knock-out tournament:
> Assuming no drawn games, determining a clear winner [requires] the same number of rounds as that of a knockout tournament [...] Compared to a knockout tournament, a Swiss system has the advantage of not eliminating anyone [and] that the final ranking gives some indication of the relative strengths of all contestants, not just of the tournament winner. [However,] sometimes a player has such a great lead that by the last round they are assured of winning the tournament [such that it] does not always end with the exciting climax
Chess tournaments I've participated in had 7 rounds. The swiss system is really cool because it is a kind of sorting algorithm, where players quickly end up confronting players their level. So if you're a beginner, you still get to play to get a better ranking (instead of "whatever, I won't win anyway").
what about trying with Shapley values?
No idea what you're talking about so I looked it up
> The Shapley value is a solution concept in cooperative game theory. It was named in honor of Lloyd Shapley, who introduced it in 1951 and won the Nobel Memorial Prize in Economic Sciences for it in 2012. To each cooperative game it assigns a unique distribution (among the players) of a total surplus generated by the coalition of all players. The Shapley value is characterized by a collection of desirable properties.
This is pretty much the whole article lede on Wikipedia but I have no idea what I just read. What's a total surplus in a cooperative game here, like how much wood your team has gathered in AoE? How does it help to "assign" that, wouldn't that rather be the result of a game? Can someone dumb this down?
Wow its very interesting, for me AI is still a big black box