Epistemic Reject Option Prediction

TL;DR

This paper introduces an epistemic reject-option predictor that abstains on uncertain inputs due to limited data, redefining optimality via Bayesian regret minimization.

Contribution

It presents a novel framework for abstaining based on epistemic uncertainty, addressing limitations of traditional approaches that focus only on aleatoric uncertainty.

Findings

First principled approach to epistemic uncertainty-based abstention.

Redefines optimal predictor as one minimizing Bayesian regret.

Enables identification of inputs with insufficient training data.

Abstract

In high-stakes applications, predictive models must not only produce accurate predictions but also quantify and communicate their uncertainty. Reject-option prediction addresses this by allowing the model to abstain when prediction uncertainty is high. Traditional reject-option approaches focus solely on aleatoric uncertainty, an assumption valid only when large training data makes the epistemic uncertainty negligible. However, in many practical scenarios, limited data makes this assumption unrealistic. This paper introduces the epistemic reject-option predictor, which abstains in regions of high epistemic uncertainty caused by insufficient data. Building on Bayesian learning, we redefine the optimal predictor as the one that minimizes expected regret -- the performance gap between the learned model and the Bayes-optimal predictor with full knowledge of the data distribution. The model abstains when the regret for a given input exceeds a specified rejection cost. To our knowledge, this is the first principled framework that enables learning predictors capable of identifying inputs for which the available training data is insufficient to support well-informed predictions.