Evaluating real-time probabilistic forecasts with application to National Basketball Association outcome prediction

TL;DR

This paper introduces new tools for evaluating real-time probabilistic forecasts, specifically applied to NBA game outcome predictions, assessing calibration and skill of different forecasting methods through simulations and real-world ESPN data.

Contribution

It develops calibration surface plots and statistical tests for assessing the quality and comparative skill of continuously updated probabilistic forecasts.

Findings

ESPN forecasts are well-calibrated.

Forecasts outperform naive models.

No significant improvement over simple logistic regression.

Abstract

Motivated by the goal of evaluating real-time forecasts of home team win probabilities in the National Basketball Association, we develop new tools for measuring the quality of continuously updated probabilistic forecasts. This includes introducing calibration surface plots, and simple graphical summaries of them, to evaluate at a glance whether a given continuously updated probability forecasting method is well-calibrated, as well as developing statistical tests and graphical tools to evaluate the skill, or relative performance, of two competing continuously updated forecasting methods. These tools are studied by means of a Monte Carlo simulation study of simulated basketball games, and demonstrated in an application to evaluate the continuously updated forecasts published by the United States-based multinational sports network ESPN on its principle webpage {\tt espn.com}. This application lends statistical evidence that the forecasts published there are well-calibrated, and exhibit improved skill over several na\"ive models, but do not demonstrate significantly improved skill over simple logistic regression models based solely on a measurement of each teams' relative strength, and the evolving score difference throughout the game.