How do you know if a football prediction model is any good?
Eleven predictors, 5,258 Scottish Premiership matches. Picking the most winners is the wrong test. Log loss, Brier score and calibration tell you which models actually understand football, and the bookmaker still beats them all.
Intermediate
Contents
The research question
If one model says Celtic have a 70% chance of winning and another says 55%, and Celtic win, which model was better?
Both "got it right". Yet one of them understood the match better than the other. This piece asks how to measure that, using a real test: eleven predictors, scored on every Scottish Premiership match they had never seen.
The dataset
| Competition | Scottish Premiership |
| Matches scored | 5,258, the same set for every predictor |
| Method | Walk-forward: each season is predicted using only earlier seasons |
| Predictors | 10 models and benchmarks, plus the bookmaker's prices |
| Figures as at | 27 September 2026 |
The predictors range from the deliberately dumb (always back the home side; use the league's historical home, draw and away rates) through Elo, Poisson and Dixon-Coles, to logistic regression, gradient boosting and an ensemble. The bookmaker's prices, with their margin removed, sit on the same board as a benchmark.
Why "how many did it get right?" is the wrong test
Accuracy counts how often a model's favourite outcome happened. It sounds sensible. It is badly misleading, for two reasons.
It throws away the probabilities. A model that says 51% home win and one that says 90% both "pick the home side". Accuracy scores them the same. But if home sides win about 51% of such matches, the first model is telling the truth and the second is wildly overconfident.
It can't tell a thoughtful model from a lazy one. Two predictors on the board always pick a home win: "always home", which puts 100% on the home side every time, and the baseline, which uses the league's long-run rates (44% home, 24% draw, 33% away) for every match. Both score 43.7% accuracy, identical. As forecasters they are nothing alike. "Always home" is certain every week, and wrong more often than not.
The method: three better scores
Log loss
Log loss looks only at the probability the model gave to what actually happened, and punishes low probabilities harshly.
$$\text{log loss} = -\ln(p_{\text{actual}})$$
In plain football
- \(p_{\text{actual}}\) is the chance the model gave to the result that happened.
- Said 50% and it happened: the score is 0.69. Said 90%: 0.11. Said 5%: 3.00. Lower is better.
- Said 0% and it happened: the score is infinite. That is why "always home" has no log loss at all. The first away win breaks it.
- A model that knows nothing and says a third each scores 1.10 every match. Anything useful must beat that.
Brier score
The Brier score compares all three probabilities with what happened, and adds up the squared gaps.
$$\begin{aligned} \text{Brier} = \;&(p_H - o_H)^2 + (p_D - o_D)^2 \\ &+ (p_A - o_A)^2 \end{aligned}$$
In plain football
- \(p_H, p_D, p_A\) are the model's chances for home, draw and away.
- \(o\) is 1 for the result that happened and 0 for the other two.
- Model says 50 / 25 / 25 and the home side wins: \((1 - 0.5)^2 + 0.25^2 + 0.25^2 = 0.375\). Lower is better; 0 is perfect.
Ranked probability score
Football results have an order: a draw sits between a home win and an away win. The ranked probability score (RPS) respects that, so backing a draw when the away side wins is punished less than backing the home side.
Take two forecasts that both give the away win 20%: one says 60% home and 20% draw, the other 20% home and 60% draw. The away side wins. Log loss and Brier score them identically. RPS prefers the second, 0.34 against 0.50, because it put its weight next to the result, not at the far end.
Calibration
A model is calibrated if its numbers mean what they say: of all the matches where it said 40%, about 40% happened. The expected calibration error (ECE) measures the gap.
Calibration matters, but it is not enough on its own. The baseline is almost perfectly calibrated, with errors of 0.001 to 0.005, because it uses long-run averages. It is also nearly useless, because it gives Celtic at home the same chances as Ross County at home. A good model must be calibrated and tell matches apart.
To see how the three scores judge a single forecast, try Score your own predictions: give a match your own chances and pick the result.
Results
Log loss on 5,258 matches (lower is better)
Three findings stand out.
1. The bookmaker beats every model. On log loss, Brier and RPS, the market's prices are the best forecasts on the board. The best model, an ensemble of the others, gets closest. That is a normal result, and the board publishes it rather than hiding it.
2. Accuracy and log loss disagree about who is good.
| Predictor | Rank on log loss | Rank on accuracy | Accuracy |
|---|---|---|---|
| Bookmaker | 1 | 1 | 53.6% |
| Ensemble | 2 | 2 | 53.3% |
| Adaptive Poisson | 3 | 5 | 52.8% |
| Elo | 4 | 4 | 52.9% |
| Poisson | 5 | 7 | 51.9% |
| Dixon-Coles | 6 | 8 | 51.9% |
| Logistic regression | 7 | 3 | 53.1% |
| Gradient boosting | 8 | 6 | 52.0% |
| League position | 9 | 9 | 51.5% |
| Baseline (historical H/D/A) | 10 | 10 | 43.7% |
Logistic regression is third on accuracy but seventh on log loss: it picks winners well but prices them badly. Adaptive Poisson is the other way round, fifth on accuracy and third on log loss. Judge by accuracy and you'd pick the wrong model.
3. The differences are small, and that's the point. The whole field of serious models sits between 0.955 and 0.996. Football is noisy; nobody gets anywhere near the certainty accuracy tables imply.
A warning about small samples
This season, after 36 matches, the table looks very different: Poisson and Dixon-Coles top it at 0.956, and the bookmaker is sixth at 0.980.
It means almost nothing yet. The board draws a 95% range around each figure and says so plainly: a season is a small sample, and the order "should not be read as a ranking until the bars pull apart". Over 5,258 matches the bookmaker is clearly best. A few weeks of results can't overturn that, however tempting the table looks. It is the same lesson as the 2020/21 season in the home advantage piece: one small sample can point anywhere.
Limitations
- One league. Everything here is the Scottish Premiership. A league with a different spread of team strengths could rank the models differently.
- The bookmaker has information the models don't: team news, injuries, and the betting market itself. Losing to it doesn't make a model bad; it shows what public results alone can reach.
- Figures move. The board rescores as matches are played. These are the numbers as at 27 September 2026; the live board is always current.
Conclusion
Don't judge a football model by how many winners it picks. Judge it by the probabilities it gives. Log loss, Brier score and RPS reward a model for being confident when it should be and cautious when it shouldn't, and calibration checks that its numbers mean what they say.
By those measures the Scottish Premiership board tells a clear story: well-built statistical models get close to the market, the market is still ahead, and a model's accuracy percentage tells you surprisingly little about which is which.
These probabilities are produced by statistical models and are intended for analysis and education. Football remains uncertain and model predictions will frequently be wrong.