Calibrating conditional risk

TL;DR

This paper introduces the problem of calibrating conditional risk in prediction models, analyzing its theoretical properties and practical implications in classification and regression, with applications to uncertainty-aware decision-making.

Contribution

It formalizes conditional risk calibration as a standalone problem, connects it to existing calibration tasks, and validates its importance through empirical studies in learning to defer frameworks.

Findings

Conditional risk calibration is equivalent to a standard regression task.

It relates to probability calibration but remains a distinct problem.

Empirical results demonstrate practical benefits in uncertainty-aware decision-making.

Abstract

We introduce and study the problem of calibrating conditional risk, which involves estimating the expected loss of a prediction model conditional on input features. We analyze this problem in both classification and regression settings and show that it is fundamentally equivalent to a standard regression task. For classification settings, we further establish a connection between conditional risk calibration and individual/conditional probability calibration, and develop theoretical insights for the performance metric. This reveals that while conditional risk calibration is related to existing uncertainty quantification problems, it remains a distinct and standalone machine learning problem. Empirically, we validate our theoretical findings and demonstrate the practical implications of conditional risk calibration in the learning to defer (L2D) framework. Our systematic experiments provide both qualitative and quantitative assessments, offering guidance for future research in uncertainty-aware decision-making.