Calibration Error for Decision Making

TL;DR

This paper introduces a new decision-theoretic calibration error metric, the CDL, which better aligns calibration with decision-making payoff improvements and presents an efficient online calibration algorithm achieving near-optimal results.

Contribution

The paper proposes the Calibration Decision Loss (CDL), a novel metric that directly measures calibration quality in terms of decision payoff improvements, and provides an efficient online calibration algorithm with improved theoretical guarantees.

Findings

CDL is separable from existing calibration metrics like ECE.

The proposed algorithm achieves near-optimal $O(rac{ ext{log } T}{ extsqrt{T}})$ expected CDL.

The algorithm bypasses previous lower bounds for calibration error rates.

Abstract

Calibration allows predictions to be reliably interpreted as probabilities by decision makers. We propose a decision-theoretic calibration error, the Calibration Decision Loss (CDL), defined as the maximum improvement in decision payoff obtained by calibrating the predictions, where the maximum is over all payoff-bounded decision tasks. Vanishing CDL guarantees the payoff loss from miscalibration vanishes simultaneously for all downstream decision tasks. We show separations between CDL and existing calibration error metrics, including the most well-studied metric Expected Calibration Error (ECE). Our main technical contribution is a new efficient algorithm for online calibration that achieves near-optimal $O(\frac{\log T}{\sqrt{T}})$ expected CDL, bypassing the $\Omega(T^{-0.472})$ lower bound for ECE by Qiao and Valiant (2021).