Conformalized Selective Regression

TL;DR

This paper introduces conformalized selective regression, combining conformal prediction with selective regression to improve uncertainty quantification and model reliability, supported by a standardized evaluation framework and extensive experiments.

Contribution

It proposes a novel conformal prediction-based approach for selective regression that accounts for model biases, along with a standardized evaluation framework.

Findings

Conformalized selective regression outperforms state-of-the-art baselines.

The approach provides reliable confidence measures for individual predictions.

The evaluation framework enables fair comparison of selective regression methods.

Abstract

Should prediction models always deliver a prediction? In the pursuit of maximum predictive performance, critical considerations of reliability and fairness are often overshadowed, particularly when it comes to the role of uncertainty. Selective regression, also known as the "reject option," allows models to abstain from predictions in cases of considerable uncertainty. Initially proposed seven decades ago, approaches to selective regression have mostly focused on distribution-based proxies for measuring uncertainty, particularly conditional variance. However, this focus neglects the significant influence of model-specific biases on a model's performance. In this paper, we propose a novel approach to selective regression by leveraging conformal prediction, which provides grounded confidence measures for individual predictions based on model-specific biases. In addition, we propose a standardized evaluation framework to allow proper comparison of selective regression approaches. Via an extensive experimental approach, we demonstrate how our proposed approach, conformalized selective regression, demonstrates an advantage over multiple state-of-the-art baselines.