TTC - Modern Draft Statistics

[zombub]

Tapping an enchantment and a sorcery, you're just trolling us now aren't you?

[Graham]

Yes?

[Jenelmo]

The enchantment could be a creature because of Opalescence.
I can't think of a way the sorcery can attack

[mtgratingtester]

So those ratings numbers are for Modern Masters?

No, it was across my whole dataset: MMA, DGM, M13, DII etc.
You can run it just for DGM once i publish it as an Excel extract if you like.

[Graham]

Oh yeah, we'd love some recent DGM numbers too.

[qrpth]

@qrpth if those are Bayesian estimates, they're credible intervals, not confidence intervals.

Thanks.

What does "Measure of being a good card: appearing in winning decks." mean, anyway?

When I wrote that comment at about 02:00 in my code I probably meant that good cards appear in good decks so let's calculate the probability of that. I think that number means nothing at all.

What was your prior distribution that gave you credible intervals extending well above 0 when the empirical evidence was Bridge From Below was 0-for-6?

P(win) = unif(0, 1) and the posterior P(win|Bridge From Below) = Beta(1, 7).

[mtgratingtester]

Does anyone know what the formula is to determine likely win% between 2 ELO's?
ie expected win% of 1800 vs 1600 is 75 etc

I have a theory that you can identify formats where luck plays a larger part because the actual win% between 2 elo's will vary more significantly from the expect win% than in formats where skill is a stronger determining factor.

[phlip]

The theory as I understand it is its a logarithmic curve - an additional 400 Elo points means you should be 10 times more good at the game. By using which it means that if a 2000-rating player plays a 1600-rating player, and they play 11 games, they should on average go 10-1. Also, going 10-1 will maintain the rating difference unchanged, so the amount the 2000-rating player gains from a win against a 1600 should be 1/10 what they drop from a loss. It's logarithmic, so a 200 rating difference should result in a win ratio of sqrt(10)-1, or pretty close to 3-1.

The theory ultimately hinges on the idea that skill levels should multiply, so if player A can beat player B 2-1, and player B can beat player C 2-1, then A should beat C 4-1, according to the theory. I honestly have no idea how well that holds up in practice...

At least, that's what I get from Wikipedia. I'm not 100% sure on the applicability to MtGO, whether it counts game wins or match wins... game wins seems more likely.

[Graham]

I don't like ELO at all
Now, I ignore mine entirely because I'm in it for the fun, but for a game like Magic when SO MUCH can be affected by luck of the draw, it just doesn't seem ideal.

Not that I have a better suggestion, but it's weird when excellent players can go so low based on just a few losses.

This has been "Graham's Random Digression Time".

[ecocd]

What does "Measure of being a good card: appearing in winning decks." mean, anyway?

When I wrote that comment at about 02:00 in my code I probably meant that good cards appear in good decks so let's calculate the probability of that. I think that number means nothing at all.

What was your prior distribution that gave you credible intervals extending well above 0 when the empirical evidence was Bridge From Below was 0-for-6?

P(win) = unif(0, 1) and the posterior P(win|Bridge From Below) = Beta(1, 7).

Thanks for the info! Non-informative prior. Cool. It's always nice finding other stat nerds. Where did you learn Bayesian Statistics? It's still a relative rare field in the U.S. outside of clinical trials. I've heard its use is more widespread in Europe.

I think one can come up with an objective measure of "good" card. I would suggest some combination of an estimate that a card ended up in a deck combined with their empirical win probability given they were played during a game. I I/you/we could come up with something, I might be willing to put my money where my mouth is

Does anyone know what the formula is to determine likely win% between 2 ELO's?

According to this random website the win probability is:

The probability that Player A would defeat Player B is is calculated by the formula:

p = win expectancy = 1/(1+10^[(Rb-Ra)/400])

A lot of organizations will tweak the formula to suit their needs, though, so they don't necessarily use the chess ELO system which would mean that's wrong.

I don't like ELO at all

WotC agrees with you as they stopped using them in any practical sense last year. It was stupid preventing good players from dropping in for pre-release, release events or FNM to protect their Grand Prix byes. I would guess it's still in use on MTGO because it's still the best system out there, flawed as it is.

Mulligan facts and strategy!
tl;dr, Assuming your opponent does not or has not yet taken a mulligan:
With a decent 2-land hand or 2-spell hand, there's no strong mathematical evidence favoring or disfavoring a mulligan from 7-to-6 so go with your gut; Only mulligan 6-to-5 if your 6 card hand is very bad

Players started with 7 cards 84% of the time, 6 cards 15% of the time, 5 cards 1% of the time and 4 cards 0.13% of the time

Players that started with a 1 card deficit after mulligans won 33% of the time (1768 out of 5333)
Players that started with a 2 card deficit after mulligans won 19% of the time (59 out of 310)
Players that started with a 3 card deficit after mulligans won 13% of the time (6 out of 47)

This actually gives us guidelines we've never had before. On average, if your opponent hasn't mulliganed and you're sitting with an iffy 7 card hand, you should keep that hand if you believe you will win more than 33% of the time with that iffy hand. You keep a 6-card hand if you think you have at least a 20% chance of winning which is very low bar.

Stated another way, relatively speaking, you have a 50% better chance of winning with an average 7 card hand than an average 6 card hand. You have a 70% better chance of winning with an average 6 card hand than an average 5 card hand. You have a 46% better chance of winning with an average 5 card hand than an average 4 card hand.

The clear breakpoint there is 6 cards to 5 cards. It has to be a very bad 6-card hand to risk 5 cards. I'm a bit surprised to learn that you should actually be more aggressive going from 5 to 4 than you are 7-to-6 and players do not recognize it either.

If we assume players are consistent in their 7-to-6 decisions, we can use that as a base for decision-making on 6-to-5 and 5-to-4. Players mulligan from 7-to-6 about 15% of the time and mulligan from 6-to-5 about 7% of the time, and 5-to-4 about 12% of the time. It looks like players, in general, are not aggressive enough going from 5-to-4.

If you want some math to justify your mulligan decisions 7-to-6 cards: Assume a 50% chance to win the game with an "average" draw. With a 2-land hand and a 17 land deck, you have a 65% chance of drawing at least 2 lands over your next 3 draws (hit every drop on the draw, only possibly miss 1 land drop on the play). With a 2-spell hand, you have a 70% chance of drawing at least 2 spells in the next 3 turns. I posit that both of those scenarios will lead to an "average" opening hand.

It's interesting to note that you have a 33% chance of having an "average" draw when starting from a 2-land hand and a 35% chance of an "average" hand with 2 spells. Those are right around the same probability of winning with an average 6-card hand. It looks like the math doesn't give any clear direction on a mulligan in those typical iffy-mulligan decisions so you're free to go with your gut. Will that hand win 1 in 3 games? If yes, keep, otherwise toss.

In the case of 6-to-5, though, you rarely want to mulligan a 2-land or 2-spell hand. Will that hand win 1 in 5 games? If yes, keep, otherwise toss. That doesn't mean you never mulligan, because there are definitely dreadful 2-land and 2-spell hands out there that won't win 1 in 5 games.

Players that chose to mulligan to 5 won 27% of the time against opponents that mulliganed to 6. If your opponent has taken a mulligan to 6, your question for 6-to-5 is whether that hand will win at least 1 in 4 games. On the flipside, if your opponent has taken a mulligan to 6, only go 7-to-6 if you feel your current 7-card hand will fail to win 1 out of 3 times against a 6-card hand.

[mtgratingtester]

The other thing i think it shows more generally is that we should be taking our mulligan decisions more seriously. It has a significant impact on our chance to win. Much more significant than say playing or drawing.

[ecocd]

Great point. Everyone "knows" about card advantage, but this is a pretty stark example of how important a single extra card can be.

[phlip]

I don't like ELO at all
Now, I ignore mine entirely because I'm in it for the fun, but for a game like Magic when SO MUCH can be affected by luck of the draw, it just doesn't seem ideal.

A ratings system like Elo can still work for high-variance games, it just really needs to be adjusted to be less responsive to individual wins/losses. For Elo, that means reducing the K-value of events by a lot. Unfortunately, the flipside of that is that it would take a lot of games for your rating to reach its correct value... so if you improve (or if you're new to MtGO but are already good enough at the game to warrant a high rating) then it will take a lot of games for that to be reflected in the rating. I haven't run the numbers, but it's probably something like: if you cut the K-values by 1/4, then the variability of the ratings will be cut by 1/2,but it'll take 4 times as many games for your rating to stabilise. For a game that many people play maybe once a week, and often less, that's probably not ideal either.

[Omnicrat]

You should DEFINITELY use those statistics in your next two LLLDrafts. One deck picking from the top of the win percentage list, another deck picking from the bottom. Given the things you guys win with, I expect the bottom deck to win at least two matches and the top deck to lose everything.

[mtgratingtester]

Updated Card Win% for DGR (+M13 +DII)
http://ge.tt/45jYIzl/v/0
Once again, overrun effects rule the roost!

I think this was asked at some point, these are from all draft formats together (Phantom + Real, Swiss, 4322 & 84)

[qrpth]

Updated Card Win% for DGR (+M13 +DII)
http://ge.tt/45jYIzl/v/0
Once again, overrun effects rule the roost!

I think this was asked at some point, these are from all draft formats together (Phantom + Real, Swiss, 4322 & 84)

Sets in that file: ALA, AVR, DGM, DKA, GTC, ISD, M13, RTR

Top 10:

Code: Select all

1: Falkenrath Noble (rarity: U, CMC: 4, set: ISD, wins: 8, losses: 0):
        win prob: 90.000;
                90% credible interval: [71.687%; 99.431%]
2: Ghoulcaller's Chant (rarity: C, CMC: 1, set: ISD, wins: 7, losses: 0):
        win prob: 88.889;
                90% credible interval: [68.767%; 99.361%]
3: Narstad Scrapper (rarity: C, CMC: 5, set: AVR, wins: 7, losses: 0):
        win prob: 88.889;
                90% credible interval: [68.767%; 99.361%]
4: Latch Seeker (rarity: U, CMC: 3, set: AVR, wins: 6, losses: 0):
        win prob: 87.500;
                90% credible interval: [65.184%; 99.269%]
5: Diregraf Captain (rarity: U, CMC: 3, set: DKA, wins: 6, losses: 0):
        win prob: 87.500;
                90% credible interval: [65.184%; 99.269%]
6: Morkrut Banshee (rarity: U, CMC: 5, set: ISD, wins: 6, losses: 0):
        win prob: 87.500;
                90% credible interval: [65.184%; 99.269%]
7: Druid's Familiar (rarity: U, CMC: 4, set: AVR, wins: 12, losses: 1):
        win prob: 86.667;
                90% credible interval: [70.326%; 97.401%]
8: Goldnight Redeemer (rarity: U, CMC: 6, set: AVR, wins: 11, losses: 1):
        win prob: 85.714;
                90% credible interval: [68.367%; 97.194%]
9: Joint Assault (rarity: C, CMC: 1, set: AVR, wins: 5, losses: 0):
        win prob: 85.714;
                90% credible interval: [60.695%; 99.150%]
10: Midnight Haunting (rarity: U, CMC: 3, set: ISD, wins: 9, losses: 1):
        win prob: 83.333;
                90% credible interval: [63.564%; 96.669%]

Bottom 10:

Code: Select all

1351: Lunar Mystic (rarity: R, CMC: 4, set: AVR, wins: 0, losses: 3):
        win prob: 20.000;
                90% credible interval: [1.274%; 52.711%]
1352: Aggravate (rarity: U, CMC: 5, set: AVR, wins: 0, losses: 3):
        win prob: 20.000;
                90% credible interval: [1.274%; 52.711%]
1353: Tormented Pariah (rarity: C, CMC: 4, set: ISD, wins: 2, losses: 12):
        win prob: 18.750;
                90% credible interval: [5.684%; 36.345%]
1354: Pithing Needle (rarity: R, CMC: 1, set: RTR, wins: 2, losses: 13):
        win prob: 17.647;
                90% credible interval: [5.315%; 34.383%]
1355: Warden of the Wall (rarity: U, CMC: 3, set: DKA, wins: 0, losses: 4):
        win prob: 16.667;
                90% credible interval: [1.021%; 45.073%]
1356: Vandalblast (rarity: U, CMC: 1, set: RTR, wins: 0, losses: 4):
        win prob: 16.667;
                90% credible interval: [1.021%; 45.073%]
1357: Ooze Flux (rarity: R, CMC: 4, set: GTC, wins: 4, losses: 25):
        win prob: 16.129;
                90% credible interval: [6.807%; 27.962%]
1358: Rest in Peace (rarity: R, CMC: 2, set: RTR, wins: 0, losses: 5):
        win prob: 14.286;
                90% credible interval: [0.850%; 39.305%]
1359: Cobbled Wings (rarity: C, CMC: 2, set: ISD, wins: 0, losses: 5):
        win prob: 14.286;
                90% credible interval: [0.850%; 39.305%]
1360: Rotting Fensnake (rarity: C, CMC: 4, set: ISD, wins: 0, losses: 5):
        win prob: 14.286;
                90% credible interval: [0.850%; 39.305%]

[mtgratingtester]

i dont think there is enough data (games) for the non-DGM sets to draw conclusions?
maybe M13 is close.

[qrpth]

i dont think there is enough data (games) for the non-DGM sets to draw conclusions?
maybe M13 is close.

The following lists have cards that appeared in 50 or more games. Top 10:

Code: Select all

26: Teleportal (rarity: U, CMC: 2, set: RTR, wins: 141, losses: 36):
        win prob: 79.330;
                90% credible interval: [74.184%; 84.102%]
58: Armed (rarity: U, CMC: 2, set: DGM, wins: 222, losses: 77):
        win prob: 74.086;
                90% credible interval: [69.850%; 78.142%]
61: Catch (rarity: R, CMC: 3, set: DGM, wins: 39, losses: 14):
        win prob: 72.727;
                90% credible interval: [62.474%; 82.033%]
64: Nefarox, Overlord of Grixis (rarity: R, CMC: 6, set: M13, wins: 49, losses: 18):
        win prob: 72.464;
                90% credible interval: [63.316%; 80.867%]
65: Gruul Ragebeast (rarity: R, CMC: 7, set: GTC, wins: 195, losses: 74):
        win prob: 72.325;
                90% credible interval: [67.769%; 76.692%]
66: Aurelia, the Warleader (rarity: M, CMC: 6, set: GTC, wins: 98, losses: 37):
        win prob: 72.263;
                90% credible interval: [65.810%; 78.343%]
67: Boros Battleshaper (rarity: R, CMC: 7, set: DGM, wins: 314, losses: 120):
        win prob: 72.248;
                90% credible interval: [68.666%; 75.713%]
68: Staff of Nin (rarity: R, CMC: 6, set: M13, wins: 43, losses: 16):
        win prob: 72.131;
                90% credible interval: [62.358%; 81.075%]
69: Angel of Serenity (rarity: M, CMC: 7, set: RTR, wins: 76, losses: 29):
        win prob: 71.963;
                90% credible interval: [64.622%; 78.835%]
70: Collective Blessing (rarity: R, CMC: 6, set: RTR, wins: 186, losses: 72):
        win prob: 71.923;
                90% credible interval: [67.253%; 76.402%]

Bottom 10:

Code: Select all

1278: Predator's Rapport (rarity: C, CMC: 3, set: GTC, wins: 18, losses: 40):
        win prob: 31.667;
                90% credible interval: [22.221%; 41.814%]
1279: Contaminated Ground (rarity: C, CMC: 2, set: GTC, wins: 43, losses: 95):
        win prob: 31.429;
                90% credible interval: [25.151%; 38.011%]
1286: Chronic Flooding (rarity: C, CMC: 2, set: RTR, wins: 39, losses: 91):
        win prob: 30.303;
                90% credible interval: [23.918%; 37.029%]
1287: Fog (rarity: C, CMC: 1, set: M13, wins: 22, losses: 52):
        win prob: 30.263;
                90% credible interval: [21.953%; 39.171%]
1288: Crypt Incursion (rarity: C, CMC: 3, set: DGM, wins: 31, losses: 73):
        win prob: 30.189;
                90% credible interval: [23.103%; 37.700%]
1293: Rakdos Ringleader (rarity: U, CMC: 6, set: RTR, wins: 16, losses: 41):
        win prob: 28.814;
                90% credible interval: [19.615%; 38.835%]
1305: Oak Street Innkeeper (rarity: U, CMC: 3, set: RTR, wins: 31, losses: 82):
        win prob: 27.826;
                90% credible interval: [21.205%; 34.886%]
1306: Restore the Peace (rarity: U, CMC: 3, set: DGM, wins: 28, losses: 76):
        win prob: 27.358;
                90% credible interval: [20.518%; 34.688%]
1313: Riot Control (rarity: C, CMC: 3, set: DGM, wins: 61, losses: 174):
        win prob: 26.160;
                90% credible interval: [21.590%; 30.959%]
1316: Merfolk of the Pearl Trident (rarity: C, CMC: 1, set: M13, wins: 15, losses: 46):
        win prob: 25.397;
                90% credible interval: [16.905%; 34.782%]

Where did you learn Bayesian Statistics? It's still a relative rare field in the U.S. outside of clinical trials. I've heard its use is more widespread in Europe.

It was mentioned in a machine learning course on Coursera. The course materials didn't have PDF estimation so I looked it up on Wikipedia.

[ecocd]

i dont think there is enough data (games) for the non-DGM sets to draw conclusions?
maybe M13 is close.

qrpth is providing credible intervals which take into account sample size in doing their estimations. You'll notice that some are certainly wider than others so conclusions can be drawn even with relatively small sample sizes assuming you're willing to use a margin of error of +-5% or +-10% or whatever.

Just remember that even a 90% credible interval is "only" going to be correct 90% of the time. If you're looking at the evaluations for 100 cards, about 10 of the true win probabilities are going to fall outside of the credible interval. They're great for evaluating 1 or 2 cards, but you have to move into the 99% credible interval range to be able to look at hundreds of cards. 99% credible intervals are often too wide to be of any practical use, though, so there's no free lunch.

[korvys]

It was mentioned in a machine learning course on Coursera.

High-five for Coursera.

[ecocd]

mtgratingtester, any chance you can provide the same spell distribution vs. W/L table information for DGM - was called draftgamedbMM.xlsx for MMA drafts? It would give an estimate on the number of spells played per game. Just to make sure, is the DGM file only from DGM/GTC/RTR drafts?

We might be able to estimate how often a card made someone's deck. With large enough samples, like the MMA dataset, we can justify some strong assumptions. First off, we can assume players had the same opportunity to draft any given card according to their printed distribution. We know WotC printed 101 commons, 60 uncommons, 53 rares and 15 mythics in MMA. Each player has an equal opportunity to draft any given card and hence each card has an equal opportunity (relative to their print distribution) to appear in a deck.

For MMA, players cast an average of about 11 spells per game. If we assume players cast about 90% of the spells they draw in a game (i.e., leaving an average of a little over 1 spell uncast when the game ends), then we can extrapolate the number of times a spell wasn't drawn or wasn't cast. In fact, it's just (23 / 11) * (100 / 90) * # played in a game. A duplicate spell modifier should probably be added for uncommons and commons or they will be understated. For MMA those will be very high (3 copies of the same booster) and for DGM, it will be pretty low (only 1 booster of each set).

Using the equal opportunity concept above, we can take the extrapolated number of times a spell appeared in a deck divided by the estimated number of times a card was drafted by a player to get the proportion of time a card was drafted and ended up in a deck.

There is a lot of guesswork in there and there are a few problems (e.g., you keep Teleportal in your hand unless playing it results in a win), but it's a far better estimate than anyone has ever been able to get before thanks to mtgratingtester. I'm hoping this is the kind of insight and excitement you were hoping to see by freely releasing these data.

I think it's child's play from there to come up with a way to combine "Playability" with "Winningness" and add a little human intelligence to take into account special cases like Overrun effects and you have an empirically-based limited power ranking.

Of course, once you have that put together, you'll realize that it's just a list of bombs at the top, bad cards at the bottom and anyone could've come up with it on their own. Still, hurray numbers!

[ecocd]

Alas, upon further work, the dataset is not built to do what I want it to do to get a playability number. In particular, there's too much double-counting. For instance, we know that any time a Sword of Light and Shadow won a game, that deck played at least 1 other game and there is around a 40% chance it was drawn and played in that game (win or lose)... except we don't know that, because there are 15,000 matches where we only have one game of information. Beyond that, there's some self-selection in that a card that wins more often is played more often simply because it ends up in more matches in 8-4 queues. There's really no way to estimate how many times a card ended up in a deck.

We can draw conclusions on the game-level, but not on the match-level or draft-level so it's still a fantastic dataset and there is/was a lot of information to be gleaned out of it. Thanks again, mtgratingtester for sharing!