Variational Learning Induces Adaptive Label Smoothing

TL;DR

This paper demonstrates that variational learning naturally produces an adaptive label smoothing mechanism that improves handling of noisy labels and distribution shifts, offering a Bayesian perspective on overconfidence in predictions.

Contribution

It introduces a variational algorithm called IVON that outperforms traditional label smoothing and aligns with existing adaptive strategies, connecting Bayesian methods to label smoothing.

Findings

IVON outperforms traditional label smoothing in experiments.

Variational learning induces effective adaptive label smoothing.

The approach offers a Bayesian perspective on managing overconfidence.

Abstract

We show that variational learning naturally induces an adaptive label smoothing where label noise is specialized for each example. Such label-smoothing is useful to handle examples with labeling errors and distribution shifts, but designing a good adaptivity strategy is not always easy. We propose to skip this step and simply use the natural adaptivity induced during the optimization of a variational objective. We show empirical results where a variational algorithm called IVON outperforms traditional label smoothing and yields adaptivity strategies similar to those of an existing approach. By connecting Bayesian methods to label smoothing, our work provides a new way to handle overconfident predictions.