How does Early Stopping Help Generalization against Label Noise?

TL;DR

This paper introduces Prestopping, a two-phase training method that uses early stopping and a safe set of samples to improve neural network generalization in the presence of label noise, outperforming existing methods.

Contribution

The paper proposes Prestopping, a novel two-phase training approach combining early stopping and safe set maintenance to effectively combat label noise in deep learning.

Findings

Outperforms four state-of-the-art methods in test error by 0.4-8.2 percentage points.

Effectively handles real-world label noise across multiple image datasets.

Demonstrates robustness of the method in practical noisy label scenarios.

Abstract

Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping, realizes noise-free training under any type of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-art methods in test error by 0.4-8.2 percent points under existence of real-world noise.