Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

TL;DR

This paper introduces a decomposition framework for probabilistic scores that explicitly separates reliability, information loss, and residual uncertainty, providing new insights into calibration and model aggregation.

Contribution

It develops decomposition identities for proper losses, linking calibration, information gain, and uncertainty, with applications to recalibration and model combination.

Findings

Decomposition identities for proper losses elucidate calibration and uncertainty.

Chain decomposition quantifies information gain between feature levels.

Explicit forms for Brier and log-loss in model aggregation and recalibration.

Abstract

Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level $\mathcal A$, the expected loss of an $\mathcal A$-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels $\mathcal A\subseteq\mathcal B$, a chain decomposition quantifies the information gain from $\mathcal A$ to $\mathcal B$. Applied to classification with features $\boldsymbol{X}$ and score $S=s(\boldsymbol{X})$, this yields a three-term identity: miscalibration, a {\em grouping} term measuring information loss from $\boldsymbol{X}$ to $S$, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.