CBMA: Improving conformal prediction through Bayesian model averaging

TL;DR

This paper introduces CBMA, a hybrid conformal prediction method combining Bayesian model averaging to enhance robustness and efficiency in predictive inference, especially under model misspecification.

Contribution

It proposes a novel Bayesian model averaging approach integrated with conformal prediction, ensuring asymptotic optimality even when models are misspecified.

Findings

Prediction sets converge to optimal efficiency if the true model is included.

CBMA outperforms traditional conformal methods under model uncertainty.

Theoretical guarantees of convergence and robustness are established.

Abstract

Conformal prediction has emerged as a popular technique for facilitating valid predictive inference across a spectrum of machine learning models, under minimal assumption of exchangeability. Recently, Hoff (2023) showed that full conformal Bayes provides the most efficient prediction sets (smallest by expected volume) among all prediction sets that are valid at the $(1 - \alpha)$ level if the model is correctly specified. However, a critical issue arises when the Bayesian model itself may be mis-specified, resulting in prediction set that might be suboptimal, even though it still enjoys the frequentist coverage guarantee. To address this limitation, we propose an innovative solution that combines Bayesian model averaging (BMA) with conformal prediction. This hybrid not only leverages the strengths of Bayesian conformal prediction but also introduces a layer of robustness through model averaging. Theoretically, we prove that the resulting prediction set will converge to the optimal level of efficiency, if the true model is included among the candidate models. This assurance of optimality, even under potential model uncertainty, provides a significant improvement over existing methods, ensuring more reliable and precise uncertainty quantification.