A Connection Between Learning to Reject and Bhattacharyya Divergences

TL;DR

This paper explores the connection between learning to reject in classification and Bhattacharyya divergences, revealing a new link with thresholding of divergence measures for improved rejection strategies.

Contribution

It introduces a novel approach linking rejection to Bhattacharyya divergence by considering joint ideal distributions, contrasting with traditional KL-based methods.

Findings

Rejecting via Bhattacharyya divergence is less aggressive than Chow's Rule.

The joint ideal distribution approach relates rejection to divergence thresholding.

A new theoretical connection between rejection and Bhattacharyya divergence is established.

Abstract

Learning to reject provide a learning paradigm which allows for our models to abstain from making predictions. One way to learn the rejector is to learn an ideal marginal distribution (w.r.t. the input domain) - which characterizes a hypothetical best marginal distribution - and compares it to the true marginal distribution via a density ratio. In this paper, we consider learning a joint ideal distribution over both inputs and labels; and develop a link between rejection and thresholding different statistical divergences. We further find that when one considers a variant of the log-loss, the rejector obtained by considering the joint ideal distribution corresponds to the thresholding of the skewed Bhattacharyya divergence between class-probabilities. This is in contrast to the marginal case - that is equivalent to a typical characterization of optimal rejection, Chow's Rule - which corresponds to a thresholding of the Kullback-Leibler divergence. In general, we find that rejecting via a Bhattacharyya divergence is less aggressive than Chow's Rule.