Machine Learning with a Reject Option: A survey

TL;DR

This survey reviews the development, formalization, and evaluation of machine learning models that can abstain from predictions to avoid errors, highlighting their architectures, learning techniques, and applications.

Contribution

It provides a comprehensive overview of the formal conditions, evaluation strategies, architectures, and learning methods for machine learning models with rejection options.

Findings

Formalization of ambiguity and novelty rejection conditions

Categorization of evaluation strategies for predictive and rejective quality

Description of architectures and standard learning techniques for rejection models

Abstract

Machine learning models always make a prediction, even when it is likely to be inaccurate. This behavior should be avoided in many decision support applications, where mistakes can have severe consequences. Albeit already studied in 1970, machine learning with rejection recently gained interest. This machine learning subfield enables machine learning models to abstain from making a prediction when likely to make a mistake. This survey aims to provide an overview on machine learning with rejection. We introduce the conditions leading to two types of rejection, ambiguity and novelty rejection, which we carefully formalize. Moreover, we review and categorize strategies to evaluate a model's predictive and rejective quality. Additionally, we define the existing architectures for models with rejection and describe the standard techniques for learning such models. Finally, we provide examples of relevant application domains and show how machine learning with rejection relates to other machine learning research areas.