Know When to Abstain: Optimal Selective Classification with Likelihood Ratios

TL;DR

This paper introduces a likelihood ratio-based approach for optimal selective classification, especially effective under covariate shift, improving reliability by abstaining from uncertain predictions across vision and language tasks.

Contribution

It applies the Neyman--Pearson lemma to develop new selection functions for selective classification, unifying existing methods and enhancing performance under covariate shift.

Findings

Likelihood ratio-based methods outperform baselines

Effective under covariate shift in vision and language tasks

Provides a unified framework for selective classification

Abstract

Selective classification enhances the reliability of predictive models by allowing them to abstain from making uncertain predictions. In this work, we revisit the design of optimal selection functions through the lens of the Neyman--Pearson lemma, a classical result in statistics that characterizes the optimal rejection rule as a likelihood ratio test. We show that this perspective not only unifies the behavior of several post-hoc selection baselines, but also motivates new approaches to selective classification which we propose here. A central focus of our work is the setting of covariate shift, where the input distribution at test time differs from that at training. This realistic and challenging scenario remains relatively underexplored in the context of selective classification. We evaluate our proposed methods across a range of vision and language tasks, including both supervised learning and vision-language models. Our experiments demonstrate that our Neyman--Pearson-informed methods consistently outperform existing baselines, indicating that likelihood ratio-based selection offers a robust mechanism for improving selective classification under covariate shifts. Our code is publicly available at https://github.com/clear-nus/sc-likelihood-ratios.