Statistical Inference for Score Decompositions

TL;DR

This paper develops inference methods for score decompositions that break down predictive scoring functions into miscalibration, discrimination, and uncertainty, applicable to various forecast types, and demonstrates their practical utility in finance and survey forecasting.

Contribution

It introduces a novel inference framework for score decompositions using linear recalibration, applicable to smooth and non-smooth scoring functions, connecting to classical calibration tests and improving forecast evaluation.

Findings

Discrimination can differ significantly even when overall predictive ability is similar.

The method provides deeper insights into forecast calibration and information content.

Exposes shortcomings in current banking regulation through forecast analysis.

Abstract

We introduce inference methods for score decompositions, which partition scoring functions for predictive assessment into three interpretable components: miscalibration, discrimination, and uncertainty. Our estimation and inference relies on a linear recalibration of the forecasts, which is applicable to general multi-step ahead point forecasts such as means and quantiles due to its validity for both smooth and non-smooth scoring functions. This approach ensures desirable finite-sample properties, enables asymptotic inference, and establishes a direct connection to the classical Mincer-Zarnowitz regression. The resulting inference framework facilitates tests for equal forecast calibration or discrimination, which yield three key advantages. They enhance the information content of predictive ability tests by decomposing scores, deliver higher statistical power in certain scenarios, and formally connect scoring-function-based evaluation to traditional calibration tests, such as financial backtests. Applications demonstrate the method's utility. We find that for survey inflation forecasts, discrimination abilities can differ significantly even when overall predictive ability does not. In an application to financial risk models, our tests provide deeper insights into the calibration and information content of volatility and Value-at-Risk forecasts. By disentangling forecast accuracy from backtest performance, the method exposes critical shortcomings in current banking regulation.