Rethinking Soft Label in Label Distribution Learning Perspective

TL;DR

This paper explores how label distribution learning (LDL) can improve CNN calibration and generalization, addressing overconfidence issues by aligning training supervision with inference criteria, and demonstrates its effectiveness across multiple datasets.

Contribution

It introduces LDL as a method to enhance CNN calibration and generalization, especially with recent data augmentation and online LDL techniques, providing a new perspective on supervision criteria.

Findings

LDL improves model calibration and generalization performance.

State-of-the-art knowledge distillation methods hinder calibration.

Online LDL offers additional benefits in large models and long training.

Abstract

The primary goal of training in early convolutional neural networks (CNN) is the higher generalization performance of the model. However, as the expected calibration error (ECE), which quantifies the explanatory power of model inference, was recently introduced, research on training models that can be explained is in progress. We hypothesized that a gap in supervision criteria during training and inference leads to overconfidence, and investigated that performing label distribution learning (LDL) would enhance the model calibration in CNN training. To verify this assumption, we used a simple LDL setting with recent data augmentation techniques. Based on a series of experiments, the following results are obtained: 1) State-of-the-art KD methods significantly impede model calibration. 2) Training using LDL with recent data augmentation can have excellent effects on model calibration and even in generalization performance. 3) Online LDL brings additional improvements in model calibration and accuracy with long training, especially in large-size models. Using the proposed approach, we simultaneously achieved a lower ECE and higher generalization performance for the image classification datasets CIFAR10, 100, STL10, and ImageNet. We performed several visualizations and analyses and witnessed several interesting behaviors in CNN training with the LDL.