The Utility of Abstaining in Binary Classification

TL;DR

This paper examines binary classifiers that can abstain from predicting to reduce errors, highlighting theoretical insights, algorithms, and connections to active learning in high-stakes applications.

Contribution

It reviews recent theoretical advances, algorithms, and fundamental tradeoffs in abstaining classifiers, emphasizing their potential to improve decision-making in critical domains.

Findings

Allowing abstentions can lead to zero-error learning in some cases.

There is a fundamental tradeoff between abstention frequency and prediction accuracy.

Efficient algorithms with provable guarantees are available for abstaining classifiers.

Abstract

We explore the problem of binary classification in machine learning, with a twist - the classifier is allowed to abstain on any datum, professing ignorance about the true class label without committing to any prediction. This is directly motivated by applications like medical diagnosis and fraud risk assessment, in which incorrect predictions have potentially calamitous consequences. We focus on a recent spate of theoretically driven work in this area that characterizes how allowing abstentions can lead to fewer errors in very general settings. Two areas are highlighted: the surprising possibility of zero-error learning, and the fundamental tradeoff between predicting sufficiently often and avoiding incorrect predictions. We review efficient algorithms with provable guarantees for each of these areas. We also discuss connections to other scenarios, notably active learning, as they suggest promising directions of further inquiry in this emerging field.