Online Selective Classification with Limited Feedback

TL;DR

This paper develops online selective classification algorithms that balance making few mistakes and minimizing abstentions in resource-limited, safety-critical settings with limited feedback, achieving tight theoretical bounds.

Contribution

It introduces simple versioning schemes for online selective classification that optimize mistake and abstention trade-offs under limited feedback and non-realisable data.

Findings

Achieves mistake bounds of T^μ with O(T^{1-μ}) abstentions.

Proves the tightness of the mistake-abstention trade-off.

Demonstrates effectiveness through experiments on realistic datasets.

Abstract

Motivated by applications to resource-limited and safety-critical domains, we study selective classification in the online learning model, wherein a predictor may abstain from classifying an instance. For example, this may model an adaptive decision to invoke more resources on this instance. Two salient aspects of the setting we consider are that the data may be non-realisable, due to which abstention may be a valid long-term action, and that feedback is only received when the learner abstains, which models the fact that reliable labels are only available when the resource intensive processing is invoked. Within this framework, we explore strategies that make few mistakes, while not abstaining too many times more than the best-in-hindsight error-free classifier from a given class. That is, the one that makes no mistakes, while abstaining the fewest number of times. We construct simple versioning-based schemes for any $\mu \in (0,1],$ that make most $T^\mu$ mistakes while incurring \smash{$\tilde{O}(T^{1-\mu})$} excess abstention against adaptive adversaries. We further show that this dependence on $T$ is tight, and provide illustrative experiments on realistic datasets.