An Exploration into why Output Regularization Mitigates Label Noise

TL;DR

This paper investigates how output regularization in loss functions, like label smoothing and entropy, enhances robustness to label noise by making the loss symmetric as the regularization strength increases, providing a theoretical understanding.

Contribution

It provides a mathematical analysis linking output regularization to symmetry in loss functions, explaining its effectiveness against label noise.

Findings

Losses with output regularization become symmetric at high regularization levels.

Regularization coefficient controls the symmetry and noise robustness of the loss.

Theoretical insights bridge practical performance and mathematical properties of noise-robust losses.

Abstract

Label noise presents a real challenge for supervised learning algorithms. Consequently, mitigating label noise has attracted immense research in recent years. Noise robust losses is one of the more promising approaches for dealing with label noise, as these methods only require changing the loss function and do not require changing the design of the classifier itself, which can be expensive in terms of development time. In this work we focus on losses that use output regularization (such as label smoothing and entropy). Although these losses perform well in practice, their ability to mitigate label noise lack mathematical rigor. In this work we aim at closing this gap by showing that losses, which incorporate an output regularization term, become symmetric as the regularization coefficient goes to infinity. We argue that the regularization coefficient can be seen as a hyper-parameter controlling the symmetricity, and thus, the noise robustness of the loss function.