Rejection via Learning Density Ratios

TL;DR

This paper introduces a novel rejection method in classification that uses density ratios between an idealized distribution and the data distribution, optimized via $ extphi$-divergence, to improve abstention decisions.

Contribution

It proposes a distributional approach to classification rejection using density ratios and $ extphi$-divergence regularization, differing from traditional loss augmentation methods.

Findings

Effective rejection decisions based on density ratios.

Robust performance on both clean and noisy datasets.

Framework applicable with $ extalpha$-divergence family.

Abstract

Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. The predominant approach is to alter the supervised learning pipeline by augmenting typical loss functions, letting model rejection incur a lower loss than an incorrect prediction. Instead, we propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance. This can be formalized via the optimization of a loss's risk with a $\varphi$-divergence regularization term. Through this idealized distribution, a rejection decision can be made by utilizing the density ratio between this distribution and the data distribution. We focus on the setting where our $\varphi$-divergences are specified by the family of $\alpha$-divergence. Our framework is tested empirically over clean and noisy datasets.