Selective Prior Synchronization via SYNC Loss

TL;DR

This paper introduces the SYNC loss, a novel training method that integrates post-hoc uncertainty information into the training process of selective prediction models, improving their performance and generalization.

Contribution

The paper proposes the SYNC loss, which combines ad-hoc and post-hoc methods by incorporating the selective prior into training, advancing selective prediction techniques.

Findings

Enhanced selective prediction performance across datasets

Surpassed previous state-of-the-art benchmarks

Improved model generalization capabilities

Abstract

Prediction under uncertainty is a critical requirement for the deep neural network to succeed responsibly. This paper focuses on selective prediction, which allows DNNs to make informed decisions about when to predict or abstain based on the uncertainty level of their predictions. Current methods are either ad-hoc such as SelectiveNet, focusing on how to modify the network architecture or objective function, or post-hoc such as softmax response, achieving selective prediction through analyzing the model's probabilistic outputs. We observe that post-hoc methods implicitly generate uncertainty information, termed the selective prior, which has traditionally been used only during inference. We argue that the selective prior provided by the selection mechanism is equally vital during the training stage. Therefore, we propose the SYNC loss which introduces a novel integration of ad-hoc and post-hoc method. Specifically, our approach incorporates the softmax response into the training process of SelectiveNet, enhancing its selective prediction capabilities by examining the selective prior. Evaluated across various datasets, including CIFAR-100, ImageNet-100, and Stanford Cars, our method not only enhances the model's generalization capabilities but also surpasses previous works in selective prediction performance, and sets new benchmarks for state-of-the-art performance.