Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation

TL;DR

This paper introduces a novel multivariate score-based conformal prediction framework for ensembles, enhancing the efficiency of uncertainty estimation in classification and regression tasks without distributional assumptions.

Contribution

It extends scalar conformal scores to multivariate scores, improving aggregation efficiency and reducing conservatism in ensemble uncertainty estimation.

Findings

Empirically outperforms existing conformal aggregation methods

Enhances prediction region efficiency in classification tasks

Improves downstream decision-making in regression settings

Abstract

Distribution-free uncertainty estimation for ensemble methods is increasingly desirable due to the widening deployment of multi-modal black-box predictive models. Conformal prediction is one approach that avoids such distributional assumptions. Methods for conformal aggregation have in turn been proposed for ensembled prediction, where the prediction regions of individual models are merged as to retain coverage guarantees while minimizing conservatism. Merging the prediction regions directly, however, sacrifices structures present in the conformal scores that can further reduce conservatism. We, therefore, propose a novel framework that extends the standard scalar formulation of a score function to a multivariate score that produces more efficient prediction regions. We then demonstrate that such a framework can be efficiently leveraged in both classification and predict-then-optimize regression settings downstream and empirically show the advantage over alternate conformal aggregation methods.