Conformal Prediction as Bayesian Quadrature

TL;DR

This paper reinterprets conformal prediction within a Bayesian framework, proposing a Bayesian quadrature approach that offers more interpretable and richer uncertainty guarantees for high-stakes machine learning applications.

Contribution

It introduces a Bayesian perspective to conformal prediction and develops a Bayesian quadrature method that enhances uncertainty quantification in predictive models.

Findings

Provides a Bayesian reinterpretation of conformal prediction

Develops a Bayesian quadrature-based uncertainty quantification method

Offers more interpretable and comprehensive guarantees for model performance

Abstract

As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive models will perform upon deployment. Distribution-free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from a Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We propose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers a richer representation of the likely range of losses to be observed at test time.