FiveThirtyEight's RAPTOR Model is not Calibrated
What is Model Calibration? Why Does it Matter?
In this post, we’re taking a step back from our usual models. Instead of trying to understand basketball with probabilistic models and accounting for our uncertainty in those models, we’re going to inspect FiveThrityEight’s RAPTOR model.
As an illustration of the difference between RAPTOR and BinomialBasketball models, RAPTOR gives the Heat a 78% chance of beating the Knicks. But how certain are they about this number? Could it be 80%? 50%? 98%? They only give us a single number and no indication if the model is extremely certain or has no idea.
What is Model Calibration?
If your model is calibrated, it simply means that when you say the Heat have a 78% chance of beating the Knicks, then you will be right 78% of the time. Of course, the game will only happen once and the Heat will either win or lose. So you have to look at all of your games you predict a team to have a 78% chance of winning, and you should end up being right 78% of the time across all of those games.
Why Does Model Calibration Matter?
If your model says the Heat have a 99.9% chance of winning tonight, you should bet on them to win, regardless of what odds you can get. The Heat are almost certain to win, so even if you get -1000 odds, you'll still come out ahead.
The problem with that strategy is if your model isn’t calibrated. What if your model says the Heat have a 99.9% chance of winning, but they’ll only actually win 55% of the time. Your model is still correct that the Heat are the likely winner, but now you shouldn’t take the -1000 odds on the heat.
The key point here is that you need your model to correctly pick the winner and also correctly describe how likely it is that the team will win. Otherwise, you’ll be too confident in your predictions (or not confident enough!).
RAPTOR’s Model is not Calibrated
So let’s see if RAPTOR is calibrated. When they say the home team has a 65% of winning, does the home team win 65% of the time? Here are their predictions for this season vs what has actually happened:
The quick way to assess this calibration plot is to see that nearly all the dots are below the dashed line. This indicates that the RAPTOR model incorrectly favors the home team too much. When RAPTOR says the home team is going to win by 70%, in fact, they only end up winning by 60-65%.
One explanation is maybe their model includes home court advantage calculated from previous seasons and this season’s home court advantage is less than it used to be. But this isn’t the case. Looking at all predictions since the 2019 season (when the first RAPTOR predictions are published), their model is still miscalibrated (favoring the home team more than it should).
So what’s going on? Usually when models are miscalibrated, it’s a big mess, but in this case the fix looks pretty simple. RAPTOR looks like it habitually gives the home team a higher probability of winning than it should. In factor, RAPTOR has predicted the home team will win in 70% of games, but home teams only win about 61% of the time in the NBA.
There’s no mention of how RAPTOR incorporates home court advantage in their explainer article. But we’ll try to get to the bottom of it somehow.
Looking Ahead
I took a break from our usual probabilistic models because I wanted to take a closer look at FiveThirtyEight’s models. I thought I would share what I found before going back to the usual models which I enjoy much more.
I’m doing my typical “put-on-5-pounds-in-the-winter” thing. It’s getting me thinking about modeling players’ body types. So maybe that’s where I’ll go next.
I've always wondered about RAPTOR. It has always seemed slightly off when dealing with streaky, injury prone teams like last year's Wizards. Great post!