In our previous post, we tackled our priors with a straightforward method. We won’t rehash everything here, but a key idea was that it starts to get complicated when dealing with more “abstract” priors, which we’ll overcome in this post.
Usually our posts are self-contained, but unfortunately, this one won’t make much sense without first catching up on the previous post.
Fixing priors by inspecting their downstream effects
We left off by fixing sigma_offense_bar
from having unreasonably fat tails. This is a parameter that says how much variance in offensive strength there is across different teams.
But now the question is: is the refined prior reasonable? It tapers off around 20. But what does 20 even mean? Should it taper off around 100?
The parameter is too abstract to have any intuition about it directly, but we can inspect the downstream consequences of its distribution. Specifically, we can look at the resulting distribution of team offensive strengths (in this case, points per game) we see using this prior distribution.
So basically, teams are constrained between averaging 85 points per game and 135 points per game. That’s actually not too bad, but it’s a bit too restrictive, right? Let’s not rule out that a team might be truly awful. It’s possible, right?
If we give sigma_offense_bar
a bit more breathing room, we end up with this refined distribution for team offensive strength:
Pretty similar, but I’m a bit more comfortable having some extreme values be possible.
As a reminder, we aren’t trying to perfectly predict team offensive strengths. We’re trying to define reasonable bounds and have 99% of our prior distribution in reasonable values.
Pushing Priors all the way down
We can keep running through this pushdown strategy for all parameter prior distributions. For example: sigma_points is a parameter that controls how much variance there is in points scored for a team given everything we know about the team’s offense and the opposing team’s defense.
I couldn’t tell you what a reasonable prior model on that parameter is, but what I can tell you is if the resulting implication of that prior makes any sense. For example, we can simulate a season of games under a prior model and look at the resulting distribution of points scored per team per game:
Maybe that could use a bit more breathing room? I think it’s pretty good.
As a quick recap:
For straightforward priors that you have direct domain knowledge on: put an uniformative prior with 99% of density within bounds of what you deem possible.
For abstract priors: evaluate the downstream consequences. Again, put 99% of the density of the downstream consequences within what you deem possible.
Looking ahead
I wanted to keep working on this model iteratively. But too many people have told me it’s going too slowly. So I think I’ll mix it up and show some other models while working through the same types of iterations.
Stan Model
// Heirarchical IRT regression
//
// This models the points of home and away teams
// as a function of the latent offensive and defensive
// strength of the teams.
//
// Specifically, this version tries to improve on the
// prior modeling.
data {
// Number of games
int<lower=1> N_games;
// Number of teams in the league
int<lower=1> N_teams;
// Home and away points scored in each game
array[N_games] int<lower=0> home_points;
array[N_games] int<lower=0> away_points;
// Team index for each game
array[N_games] int<lower=1, upper=N_teams> home_team;
array[N_games] int<lower=1, upper=N_teams> away_team;
// Threshold for home field advantage
// 99% of the density should be between 0 and this value
real home_field_advantage_threshold;
}
transformed data {
// Section 3.3.1 https://betanalpha.github.io/assets/case_studies/prior_modeling.html
real home_field_advantage_prior_sigma = home_field_advantage_threshold / 2.57;
}
generated quantities {
// Prior Modeling
// Average strength of the teams
real theta_offense_bar;
theta_offense_bar = normal_rng(110, 6.67);
// Home field advantage
// Put 99% of dennsity between 0 and input {home_field_advantage_threshold}
real <lower=0> home_field_advantage;
home_field_advantage = abs(normal_rng(0, home_field_advantage_prior_sigma));
// Variations of the teams strength
real<lower=0> sigma_offense_bar;
real<lower=0> sigma_defense_bar;
sigma_offense_bar = abs(normal_rng(0, 12));
sigma_defense_bar = abs(normal_rng(0, 12));
// Individual team strength
real theta_offense;
real theta_defense;
theta_offense = normal_rng(theta_offense_bar, sigma_offense_bar);
theta_defense = normal_rng(0, sigma_defense_bar);
// Gaussian noise in the points
real<lower=0> sigma_points;
sigma_points = abs(normal_rng(0, 7));
real home_points_regression = home_field_advantage + theta_offense + theta_defense;
real away_points_regression = theta_offense + theta_defense;
real home_points_sample = normal_rng(home_points_regression, sigma_points);
real away_points_sample = normal_rng(away_points_regression, sigma_points);
}