Selective Prediction via Training Dynamics

TL;DR

This paper introduces a training dynamics-based framework for selective prediction that improves accuracy and utility trade-offs without modifying model architecture or training objectives, applicable across various domains.

Contribution

The authors propose a novel, domain-agnostic method using training dynamics to enhance selective prediction performance without altering existing training procedures.

Findings

Outperforms state-of-the-art methods on image classification benchmarks.

Effective across classification, regression, and time series tasks.

Does not require changes to model architecture or training process.

Abstract

Selective Prediction is the task of rejecting inputs a model would predict incorrectly on. This involves a trade-off between input space coverage (how many data points are accepted) and model utility (how good is the performance on accepted data points). Current methods for selective prediction typically impose constraints on either the model architecture or the optimization objective; this inhibits their usage in practice and introduces unknown interactions with pre-existing loss functions. In contrast to prior work, we show that state-of-the-art selective prediction performance can be attained solely from studying the (discretized) training dynamics of a model. We propose a general framework that, given a test input, monitors metrics capturing the instability of predictions from intermediate models (i.e., checkpoints) obtained during training w.r.t. the final model's prediction. In particular, we reject data points exhibiting too much disagreement with the final prediction at late stages in training. The proposed rejection mechanism is domain-agnostic (i.e., it works for both discrete and real-valued prediction) and can be flexibly combined with existing selective prediction approaches as it does not require any train-time modifications. Our experimental evaluation on image classification, regression, and time series problems shows that our method beats past state-of-the-art accuracy/utility trade-offs on typical selective prediction benchmarks.