Analytics, have some humility: a statistical view of fourth-down decision making

TL;DR

This paper emphasizes the importance of incorporating uncertainty quantification into fourth-down decision-making in American football, revealing that current estimates underestimate true uncertainty due to model limitations.

Contribution

It introduces a bootstrap-based method to quantify uncertainty in win probability estimates, highlighting the need for humility in decision-making.

Findings

Uncertainty in optimal decision estimates is much higher than previously acknowledged.

Current sports media underestimate the true uncertainty in fourth-down decisions.

Bootstrapping provides a practical way to incorporate uncertainty into decision models.

Abstract

The standard mathematical approach to fourth-down decision making in American football is to make the decision that maximizes estimated win probability. Win probability estimates arise from machine learning models fit from historical data. These models attempt to capture a nuanced relationship between a noisy binary outcome variable and game-state variables replete with interactions and non-linearities from a finite dataset of just a few thousand games. Thus, it is imperative to knit uncertainty quantification into the fourth-down decision procedure; we do so using bootstrapping. We find that uncertainty in the estimated optimal fourth-down decision is far greater than that currently expressed by sports analysts in popular sports media.