Robust Decision Making with Partially Calibrated Forecasts

TL;DR

This paper investigates how decision makers should act based on partially calibrated forecasts, establishing a minimax optimal decision rule that is computationally feasible and aligns with trusting predictions under weaker calibration conditions.

Contribution

It introduces a minimax framework for decision making with partially calibrated forecasts and characterizes optimal policies, showing trust in predictions under decision calibration.

Findings

Minimax optimal decision rule derived via duality.

Trust in predictions aligns with decision calibration, a weaker calibration notion.

Efficient algorithms for decision rules in regression models.

Abstract

Calibration has emerged as a foundational goal in ``trustworthy machine learning'', in part because of its strong decision theoretic semantics. Independent of the underlying distribution, and independent of the decision maker's utility function, calibration promises that amongst all policies mapping predictions to actions, the uniformly best policy is the one that ``trusts the predictions'' and acts as if they were correct. But this is true only of \emph{fully calibrated} forecasts, which are tractable to guarantee only for very low dimensional prediction problems. For higher dimensional prediction problems (e.g. when outcomes are multiclass), weaker forms of calibration have been studied that lack these decision theoretic properties. In this paper we study how a conservative decision maker should map predictions endowed with these weaker (``partial'') calibration guarantees to actions, in a way that is robust in a minimax sense: i.e. to maximize their expected utility in the worst case over distributions consistent with the calibration guarantees. We characterize their minimax optimal decision rule via a duality argument, and show that surprisingly, ``trusting the predictions and acting accordingly'' is recovered in this minimax sense by \emph{decision calibration} (and any strictly stronger notion of calibration), a substantially weaker and more tractable condition than full calibration. For calibration guarantees that fall short of decision calibration, the minimax optimal decision rule is still efficiently computable, and we provide an empirical evaluation of a natural one that applies to any regression model solved to optimize squared error.