Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction

TL;DR

This paper formalizes the use of conformal prediction for binary classification with reject options, providing distribution-free error guarantees and practical methods to balance error and reject rates.

Contribution

It introduces a formal framework connecting conformal prediction with reject options, deriving theoretical error guarantees and practical error-reject trade-off tools.

Findings

Distribution-free error guarantees for reject options

Finite sample estimates for error rates

Error-reject trade-off curves

Abstract

Machine learning (ML) models always make a prediction, even when they are likely to be wrong. This causes problems in practical applications, as we do not know if we should trust a prediction. ML with reject option addresses this issue by abstaining from making a prediction if it is likely to be incorrect. In this work, we formalise the approach to ML with reject option in binary classification, deriving theoretical guarantees on the resulting error rate. This is achieved through conformal prediction (CP), which produce prediction sets with distribution-free validity guarantees. In binary classification, CP can output prediction sets containing exactly one, two or no labels. By accepting only the singleton predictions, we turn CP into a binary classifier with reject option. Here, CP is formally put in the framework of predicting with reject option. We state and prove the resulting error rate, and give finite sample estimates. Numerical examples provide illustrations of derived error rate through several different conformal prediction settings, ranging from full conformal prediction to offline batch inductive conformal prediction. The former has a direct link to sharp validity guarantees, whereas the latter is more fuzzy in terms of validity guarantees but can be used in practice. Error-reject curves illustrate the trade-off between error rate and reject rate, and can serve to aid a user to set an acceptable error rate or reject rate in practice.