Jump balls, rating falls, and elite status: A sensitivity analysis of three quarterback rating statistics

TL;DR

This study conducts a sensitivity analysis of three NFL quarterback rating systems to understand how random game factors influence perceived quarterback performance and rankings.

Contribution

It provides a detailed sensitivity analysis of three rating systems, revealing their varying responsiveness to different game outcome factors.

Findings

Traditional rating is most sensitive to touchdowns, interceptions, and completions.

Burke rating is most sensitive to sacks.

Team passing rankings are highly affected by factors outside the quarterback's control.

Abstract

Quarterback performance can be difficult to rank, and much effort has been spent in creating new rating systems. However, the input statistics for such ratings are subject to randomness and factors outside the quarterback's control. To investigate this variance, we perform a sensitivity analysis of three quarterback rating statistics: the Traditional 1971 rating by Smith, the Burke, and the Wages of Wins ratings. The comparisons are made at the team level for the 32 NFL teams from 2002-2015, thus giving each case an even 16 games. We compute quarterback ratings for each offense with 1-5 additional touchdowns, 1-5 fewer interceptions, 1-5 additional sacks, and a 1-5 percent increase in the passing completion rate. Our sensitivity analysis provides insight into whether an elite passing team could seem mediocre or vice versa based on random outcomes. The results indicate that the Traditional rating is the most sensitive statistic with respect to touchdowns, interceptions, and completions, whereas the Burke rating is most sensitive to sacks. The analysis suggests that team passing offense rankings are highly sensitive to aspects of football that are out of the quarterback's hands (e.g., deflected passes that lead to interceptions). Thus, on the margins, we show arguments about whether a specific quarterback has entered the elite or remains mediocre are irrelevant.