Generalized Venn and Venn-Abers Calibration with Applications in Conformal Prediction

TL;DR

This paper introduces a unified calibration framework extending Venn and Venn-Abers methods to various prediction problems, providing finite-sample guarantees and capturing epistemic uncertainty, with applications in conformal prediction.

Contribution

It generalizes Venn and Venn-Abers calibration to broader loss functions and prediction tasks, including multicalibration and conformal prediction, with finite-sample guarantees.

Findings

Set-valued predictions converge to conditionally calibrated points.

Framework recovers and extends conformal prediction methods.

Provides novel prediction intervals with quantile-conditional coverage.

Abstract

Ensuring model calibration is critical for reliable prediction, yet popular distribution-free methods such as histogram binning and isotonic regression offer only asymptotic guarantees. We introduce a unified framework for Venn and Venn-Abers calibration that extends Vovk's approach beyond binary classification to a broad class of prediction problems defined by generic loss functions. Our method transforms any perfectly in-sample calibrated predictor into a set-valued predictor that, in finite samples, outputs at least one marginally calibrated point prediction. These set predictions shrink asymptotically and converge to a single conditionally calibrated prediction, capturing epistemic uncertainty. We further propose Venn multicalibration, a new approach for achieving finite-sample calibration across subpopulations. For quantile loss, our framework recovers group-conditional and multicalibrated conformal prediction as special cases and yields novel prediction intervals with quantile-conditional coverage.