Adaptive Set-Mass Calibration with Conformal Prediction

TL;DR

This paper introduces a new set-based calibration method called cumulative mass calibration, which improves probability calibration in high-risk applications by providing distribution-free guarantees and scalable post-hoc calibration techniques.

Contribution

It proposes a novel cumulative mass calibration concept, a new error measure (CMCE), and scalable post-hoc calibration methods based on conformal prediction with theoretical guarantees.

Findings

Improved calibration metrics on multi-class image benchmarks

Effective post-hoc calibration with conformal prediction

Scalable framework with theoretical guarantees

Abstract

Reliable probabilities are critical in high-risk applications, yet common calibration criteria (confidence, class-wise) are only necessary for full distributional calibration, and post-hoc methods often lack distribution-free guarantees. We propose a set-based notion of calibration, cumulative mass calibration, and a corresponding empirical error measure: the Cumulative Mass Calibration Error (CMCE). We develop a new calibration procedure that starts with conformal prediction to obtain a set of labels that gives the desired coverage. We then instantiate two simple post-hoc calibrators: a mass normalization and a temperature scaling-based rule, tuned to the conformal constraint. On multi-class image benchmarks, especially with a large number of classes, our methods consistently improve CMCE and standard metrics (ECE, cw-ECE, MCE) over baselines, delivering a practical, scalable framework with theoretical guarantees.