White Ball Analytics

MS Dhoni recently forfeited the opportunity to become the most successful captain in the Indian Premier League. Instead, with one throwback, toe-seeking yorker, Lasith Malinga handed the trophy to his own captain, Rohit Sharma of the Mumbai Indians. Four titles in seven years, all at Dhoni’s expense; it is an impressive record… yet Dhoni’s record is arguably superior when we consider the broader extent of his achievements through 12 IPL seasons: Chennai Super Kings have never failed to make the playoffs under his leadership. He has competed in 9 finals. He has taken 44 different players with him to those finals*.

The 2019 final was decided by a single run. In a match strewn with errors, just one more could have swung the result, and the accolade of most successful IPL captain could have swung from one man to another. A few breaks of fortune can win you a final, but they could never explain the extraordinary consistency shown by Dhoni, Fleming and the Chennai Super Kings since the IPL began. Besides, we’ve been watching them the whole time. We know why they win. They trust their core players. They get their tactics spot on. They…

To be honest, a lot of the stuff they do is confusing. One of the reasons people describe Dhoni as an instinctive captain is because it can be hard to understand what he is doing. His mysterious methods have inspired entire articles. Because he must be doing something right. We project an aura of invincibility around him because we assume that he knows something we don’t.

This does not need to be true. He could be lucky.

* This is a record. Interestingly, Yusuf Pathan is second on the list, playing alongside 36 other players, but for three completely different teams: Rajasthan Royals, Sunrisers Hyderabad, and Kolkata Knight Riders. Rohit Sharma has played alongside 32 other players in finals

Both captains want to win the toss. Before the match starts one enjoys the single moment where events are completely within his control – it’s his call. And having the opportunity to make that call can be hugely important: the team batting second has won 59% of matches during the last four seasons of the IPL (2016-2019); teams choose to bat second 83% of the time; the Kolkata Knight Riders have never chosen to bat first during that period (that some captains do is a mystery to me). Winning the coin toss in the IPL immediately gives your team the advantage.

Let’s briefly consider a world in which the coin toss doesn’t just give an advantage but completely determines the final outcome. Maybe, there is a pre-contest before each match. There is rock-paper-scissors contest between the Supreme Super King and the Supreme Mumbai Indian to take control over the 2019 final. It is a ritual that occurs each match, between gods of the IPL, eight minor deities who control the fate of every match, designing intricate plot twists and dramatic last-ball finishes in which their chosen team ultimately triumphs.

What would the IPL look like if every game were 50:50 and outcomes entirely random*?

The true IPL standings for 2019 compared to standings generated by the coin toss results only

We can see what the standings would look like by using the toss results for the 2019 IPL season. Rajasthan Royals managed to win 11 of 14 tosses and would have topped the league table. Meanwhile, the Royal Challengers Bangalore languish at the bottom, winning only 4 tosses**. Strangely, the ‘coin’ table has a greater point spread than the real table. Rajasthan lead outright with 22 points compared to 18 points for the best teams in real life. And Bangalore only earn 8 points, much further adrift from the playoffs.

Compare this to the English Premier League. There is no way Manchester City and Liverpool could both be so far ahead if every result was determined on the flip of a coin. Even after 14 games (the number of matches in an IPL season), those two were already leading the league, on 38 and 36 points respectively, out of a possible 42. The top five also remained fixed in place until the end of the season. Nothing like the chaos we see in the IPL each year. In almost all sports leagues, the eventual point spread is greater than if every game were a 50:50 proposition. This is because some teams are stronger than others. Their games are more like 60:40 or 70:30. We can intuit how much team strength matters based on the amount of variation in the final standings.

Conversely, the 2019 IPL season is even more random than perfectly random. There is no need to consider teams strength as an explanation for some teams finishing higher than others in the league table. We can attribute all the variation to randomness, with room to spare.

* we are discounting the possibility that the Supreme Super King (for example) is particularly adept at rock-paper-scissors and thus gets to control all the games his team are involved in
** for what its worth, with batting second conferring a 59-41 advantage, this disparity is worth about 2.5 points for Rajasthan over Bangalore

In numerous studies the stated task for observers was predicting which side of a “T-maze” held food for a subject rat. Unbeknown to both to observers and the rat, the maze was rigged such that the food was randomly placed (no pattern), but 60 percent of the time on one side and 40 percent of the time on the other. Whilst the rat quickly “gets it” and waits at the ‘60 percent’ side every time, waiting to snatch the food, human observers were right only 52 percent of the time, confounded by their tendency to identify false patterns and predict accordingly.

Our instinct is to identify patterns and build the narratives to explain them. We are hardwired to reject probabilistic conclusions that accept the inevitability of randomness and chance.

When watching sport, the empiricist in me is often winning against these instincts. I am more probabilistic rat than narrative-seeking human. The baseline is to assume sporting outcomes are mostly random and change my mind only once I have seen enough evidence to the contrary. Of all sports, cricket makes it especially easy to accept the influence of chaos over order. The randomness of T20 continually confounds my attempts to rate players – especially bowlers – or to identify weaknesses against bowling types and specific deliveries. The samples are never big enough. Players seem to contradict their historic records in every innings.

Yet the narrative of Captain Cool seems resistant to these wild fluctuations. There must be enough evidence to refute him being just another statistical anomaly, the beneficiary of small sample sizes, and our reluctance to accept the randomness of reality.

This wouldn’t be a White Ball Analytics article if I didn’t bring any hardcore maths to the debate. Hopefully, I can go so way towards quantifying the degree of luck involved in winning matches in the IPL, and the likelihood that we could see a team with Chennai’s record purely by chance.

The 2019 season provides just one datapoint. Comparing match outcomes to the real-life toss results might also be unfair. Perhaps Rajasthan and Chennai were exceptionally lucky this year (they were); normally we wouldn’t expect such coin-tossing success. We know that the standings are not normally so ‘bunched’ as they were in 2019. And we also know that the toss results can be more equitable too. So instead, I have simulated thousands of matches as though each match were decided in this way. We can then compare how ‘bunched’ the standings are with past seasons to assess the influence of luck in the final standings.

But before diving into the maths bit, let me indulge in arguing the two extreme conclusions that we might arrive at: either Chennai’s record is pure skill or Chennai’s record is pure luck…

Dhoni has led India into more matches than anybody else. He has led them to victory in all three limited-over ICC tournaments, a feat unequalled by any player from any country. And we see his impact in the IPL: bowlers perform well under him, Chennai were one of the first to flex batting line-ups to situation, and, more recently, he has transferred his uncanny review-challenging from International cricket into the IPL. He is also a great wicketkeeper-batsman. Simply having him in your team is obviously going to give you an advantage. Part of his success leading India is that he was part of the team, famously hitting the winning six in the 2011 World Cup final. His wicketkeeping is an invaluable asset for a team playing at Chepauk with world class spin bowlers.

Dhoni and Fleming have applied their high cricketing IQs to recruiting talent in auctions. They had talented squads throughout 12 seasons of the IPL: great all-rounders, the second highest run scorer in the IPL (Suresh Raina), and strong bowling line-ups. The current squad probably includes the most versatile bowling unit in the league. Chennai have backed their core players to perform again and again. They don’t chop and change like some other teams and this continuity breeds success.

Even ignoring these arguments, the statistical case is clear: it’s virtually impossible to make the playoffs in 9 consecutive seasons (ignoring their two-year hiatus) by chance alone. Half the teams make the playoffs each year. Some quick maths tells us that the odds are 512-1 against.

There are so many reasons to believe that Dhoni’s achievements are real. The only reason we can’t conclusively, statistically, prove his impact via the data is that there isn’t enough of it. Clearly, luck plays a part in T20 – Mumbai stumbling to victory in the 2019 final is more than enough proof of that – but the Chennai Super Kings have suffered just as much bad luck along with the good. It would be foolish to bet against them next season having seen what they have done so far.

On the other hand…

Yes, Dhoni is a great cricketer who would improve almost any T20 team – but this has a cost. 15 Crore is too much to pay for a player with who consistently scored below the par rate in the years before the 2018 auction. Dhoni cost his team approximately 54 runs in 75 games between 2015 and 2017, when compared to how fast an average player would have scored in similar situations. Yet the only IPL player occupying a greater share of his team’s salary cap is Virat Kohli.

And yes, the chance of any specific team making the playoffs so regularly is small, but even in a purely random process, one of the eight teams is going to be more successful than the others. Completely random sequences always house order within them somewhere, and we are hard-wired to spot that order and attach meaning to it. Chennai are the most successful team in the IPL because at least one team has to be. Meanwhile, we scramble to construct the narratives to explain why.

By most accounts, Chennai were expected to struggle following the 2018 auction. Consider Shane Watson. He was their second biggest overseas signing (after Bravo), and it remains one of the more questionable buys of the auction. Hindsight has since further undermined that decision. His knock in the 2018 final is the only redeeming feature from two mediocre seasons. His best innings have all been characterised by a staggering amount of good fortune. He was dropped three times and survived a run-out in the most recent final, yet he failed to take the opportunity to blast Mumbai out of the game. And you can’t blame team selection, per se. Chennai have no realistic alternative in their squad. The reason they exhibit such admirable continuity in their team selections is because they simply don’t have any other options.

Smart analytics are certainly not necessary to make quality decisions… but the fact that the franchise openly decries analytics and forgoes team meetings is not a ringing endorsement for an organisation with good process and standards. Dhoni’s instinctive captaincy decisions don’t always make much sense either, even in retrospect. Jadeja has batted ahead of Bravo on 42 occasions, with a strike rate of 124 in those matches, taking the required rate to almost unmanageable levels. Bowling him in the final over against Delhi was also bizarre. He was hit for 16 runs. The only reason we respect Dhoni’s decision-making so much is that when he makes bad decisions, he wins anyway. It all adds to the mystique but it’s just luck.

My guess is that most reading fall on the ‘Chennai are good’ side of the fence, rather than the ‘Chennai are lucky’ side. I was about 70:30 on that side too before writing this. The following analysis attempts to explore the statistical evidence for either side. It delves into what IPL seasons would look like, depending on whether we assume outcomes are mainly luck- or skill- driven.

I simulated every season of the IPL (2008-2019) as though each match were a 50:50 contest. I also coded the model to produce final scores, so that I could calculate NRR when needed. After simulating each season 1,000 times, I looked to see how often the simulation produced a table that was as extreme, or more extreme, than the actual points table for that year. This was measured by summing the difference between the final result and a completely balanced table (where all teams have 14 points*). I also looked at how often the luckiest team, via the simulation, gathered more points than the top-ranked team in real life, and vice versa for the unluckiest.

This method produced very little evidence that the IPL is anything more than a coin-flipping contest. Using the 5% threshold oft used throughout academia, only one real-life season produced an outcome that was significantly different to my simulations. And given that we effectively ran the same experiment 12 times, it shouldn’t be surprising to see one significant result emerge.

However, there is a feature of the actual league tables that could not be replicated by the sim. That feature concerns how the standings evolve within IPL seasons. Generally, the teams who start the fastest, rising to the top of the standings early, also finish strongly. Or, in maths speak, there is a small, positive (+15%) correlation between a team’s results in the first half of their season and their results in the second. Successful teams remain successful. This did not happen in the simulations, by design.

Moreover, the aura surrounding Dhoni and the Chennai Super Kings is founded on multiple successes throughout their history. Estimating the randomness of single seasons in isolation does not address that. It does not explain their consistent presence in playoffs. Assuming a 50% chance of making the playoffs in each season, the chances of achieving this record are tiny – 0.2%. But that is the wrong calculation. We need to consider the chances that any of the eight core teams of the IPL should be so successful; hindsight the only reason to isolate Chennai specifically, because they are the ones who have already done it. It may seem alien to think that the Delhi Daredevils might have made as many finals as Chennai have but if the league were truly random, and the two teams had their fortunes reversed, we might have constructed a different narrative in our minds.

My simulations suggested that, over the course of 12 seasons, there would almost always – 86% of the time – be one team to win the 3 or 4 titles. It also isn’t strange for two teams to have won 7 titles between them. Chennai’s playoff record does seem rather unique – very rarely in 1,000 simulations did a team make the playoffs in every single season (0.2%) – although, when a team did replicate it, it was usually Rajasthan or Chennai as they played fewer season than anyone else, making the feat slightly more feasible.

Even this may still be an illusion from randomness. Maybe we need to zoom out even further. When we considered the chance that Chennai could have been as successful as they have, purely by chance, the resulting probabilities were tiny, and so we zoomed out… we considered the chance that any team could reach the playoffs in every year. The resulting probabilities were still small, but more plausible. Additionally, our metric of choice was still not chosen independently from our experience of past events; the only reason that we are considering consecutive playoff appearances as our metric is because Chennai have already done it. Perhaps we need to be somehow computing the probability that any team might achieve any feat as impressive as Chennai’s. Whatever it may be.

* In the seasons with more than 8 teams the standings were scaled so that they could be compared side-by-side against other seasons

The first sports analytics articles I ever read were about estimating the role of luck in American football. A website called Advanced NFL Stats, founded by Brian Burke, calculated what the distribution of team win percentages would be across the league based on different combinations of luck and skill, much like I have described above. Burke took it one step further: each game was not simply decided on a coin toss, instead, there was an X% probability that the game would be decide on a coin toss, otherwise the ‘stronger’ team would always win.

“It dawned on me to create another simulation, one that synthesized the pure-luck and pure-skill distributions together in varying degrees. (10% luck/90% skill, 20% luck/80% skill, etc.) Basically, the luck% variable determined a percentage of games (chosen at random) to be decided by pure luck, essentially a coin flip. The remainder of the games were credited to the superior team.”

He determined that the best fit between his simulated results and real life happened when games where about 53% luck. Using the same approach, I make the IPL somewhere between 73-85% luck.

The most true-to-life method was to assign each team a different strength, from which I could calculate win probabilities for any match-up. In the most accurate version, the best team had a 61% chance of beating an average team. The simulations, more or less, matched both the size of the spread in the final league table and the 15% correlation between team results in the first and second halves of the season. It wasn’t quite possible to get both exactly right – either the standings were not ‘bunched’ enough, or the correlations were too low – but it was better than anything else I tried. Through 1000 simulations, the strongest team averaged 4 titles across 12 seasons, just like Mumbai.

It is worth clarifying what we define as ‘luck’. In his book on the subject, Ed Smith rightly notes that the average person, the statistician, and the gambler often mean completely different things when they refer to luck. His own definition – what happens to me that is outside my control – is one that I can broadly agree with. It makes sense and it’s a useful way to draw lines between what can and can’t be called luck.

Setting aside any philosophical debates on free will, the problem with Smith’s definition is that it depends on perspective. The pronoun, my, is key. It only works on a completely personal level. Watson was dropped three times in the 2019 final and survived a run-out before his innings had barely started. From his perspective, and CSK’s, that is lucky. His control over proceedings vanished as soon as the ball left the bat. From Mumbai’s perspective, they had the chance to dismiss him on four occasions and, in each case, the outcome rested entirely in their hands.

The statistician’s definition* is more concerned with predictable and repeatable outcomes. Malinga might have a superb yorker execution rate but no one can accurately predict when he will and won’t land them. The outcome of the 2019 final, therefore, counts as luck, to some degree. The fact that he can successfully execute 70% of his yorkers across an entire season is skill. That he executed on the most impactful ball of the IPL final is partly luck.

* The gambler’s definition is often similar to the statistician’s, but it only includes events where he loses money

It is tempting to conclude that the IPL is just a show. Perhaps the IPL gods are not flipping coins for each individual match, and then writing an engaging script to produce the necessary score line. Instead, they flip coins to decide who gets to win the title and then manufacture a league narrative that maximises excitement and tension. That would be the most reductive way to simultaneously explain the overly ‘bunched’ standings and the remarkable sustained success of Chennai – but it isn’t the only way.

The right blend of randomness can also produce a chaotic league phase whilst also allowing strong teams to demonstrate their superiority. It is almost certain that some IPL teams are better than others. The distribution of team strengths that best fits the data comprises a best team with a 61% chance over an average team. And if one team is better than the all others, it is probably Chennai, and Dhoni is probably the reason why. True outliers in sport are always a combination of skill and luck – you need both. It is unlikely that Chennai’s playoff streak happens if they were merely an average team, but it also doesn’t happen unless they receive a few lucky breaks at the right times.

Luck clearly plays a huge role in T20 cricket. In the 2019 season alone, we have seen 6 last ball finishes, last over no-balls, and a Mankad that generated a c.20% swing in win probability. Each time, the losing team was inches away from victory, or the playoffs, or the title. We constantly underestimate the effects of randomness when we assess the careers of great players and the histories of great teams. We can’t help it. It’s in a DNA. As such, I do not believe it outlandish to suggest that if the IPL started over, Dhoni would have just as much chance of a losing record as a winning one. But, if we found ourselves magically transported to such a parallel universe, I certainly wouldn’t bet against him.