Midpoint Regularization: from High Uncertainty Training to Conservative Classification

TL;DR

This paper introduces Pairwise Label Smoothing (PLS), a novel regularization technique that creates high-uncertainty midpoint samples from pairs of data points, leading to improved classification accuracy and more conservative predictions.

Contribution

The paper proposes PLS, extending label smoothing by averaging sample pairs to generate uncertain midpoints, significantly enhancing model performance and calibration.

Findings

PLS achieves up to 30% relative error reduction.

PLS produces very low softmax scores for in- and out-of-distribution samples.

PLS outperforms traditional label smoothing in empirical tests.

Abstract

Label Smoothing (LS) improves model generalization through penalizing models from generating overconfident output distributions. For each training sample the LS strategy smooths the one-hot encoded training signal by distributing its distribution mass over the non-ground truth classes. We extend this technique by considering example pairs, coined PLS. PLS first creates midpoint samples by averaging random sample pairs and then learns a smoothing distribution during training for each of these midpoint samples, resulting in midpoints with high uncertainty labels for training. We empirically show that PLS significantly outperforms LS, achieving up to 30% of relative classification error reduction. We also visualize that PLS produces very low winning softmax scores for both in and out of distribution samples.