Delving Deep into Label Smoothing

TL;DR

This paper introduces an Online Label Smoothing (OLS) method that generates more reliable soft labels based on model prediction statistics, improving classification accuracy and robustness to noisy labels in deep neural networks.

Contribution

The paper proposes a novel OLS strategy that adaptively creates soft labels from model predictions, enhancing performance and noise robustness over traditional label smoothing methods.

Findings

Improves classification accuracy on CIFAR-100, ImageNet, and fine-grained datasets.

Significantly enhances robustness of DNNs to noisy labels.

Outperforms existing label smoothing techniques in experiments.

Abstract

Label smoothing is an effective regularization tool for deep neural networks (DNNs), which generates soft labels by applying a weighted average between the uniform distribution and the hard label. It is often used to reduce the overfitting problem of training DNNs and further improve classification performance. In this paper, we aim to investigate how to generate more reliable soft labels. We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category. The proposed OLS constructs a more reasonable probability distribution between the target categories and non-target categories to supervise DNNs. Experiments demonstrate that based on the same classification models, the proposed approach can effectively improve the classification performance on CIFAR-100, ImageNet, and fine-grained datasets. Additionally, the proposed method can significantly improve the robustness of DNN models to noisy labels compared to current label smoothing approaches.