Noisy Early Stopping for Noisy Labels

TL;DR

This paper introduces a practical approach called Noisy Early Stopping (NES) that effectively prevents overfitting in neural network training with noisy labels by monitoring accuracy on noisy data instead of requiring a noise-free validation set.

Contribution

The study demonstrates that NES achieves near-optimal early stopping performance without a noise-free validation set, simplifying implementation and reducing costs in noisy label environments.

Findings

NES performs robustly across standard benchmarks.

Monitoring noisy data accuracy suffices for effective early stopping.

Theoretical analysis clarifies conditions for NES effectiveness.

Abstract

Training neural network classifiers on datasets contaminated with noisy labels significantly increases the risk of overfitting. Thus, effectively implementing Early Stopping in noisy label environments is crucial. Under ideal circumstances, Early Stopping utilises a validation set uncorrupted by label noise to effectively monitor generalisation during training. However, obtaining a noise-free validation dataset can be costly and challenging to obtain. This study establishes that, in many typical learning environments, a noise-free validation set is not necessary for effective Early Stopping. Instead, near-optimal results can be achieved by monitoring accuracy on a noisy dataset - drawn from the same distribution as the noisy training set. Referred to as `Noisy Early Stopping' (NES), this method simplifies and reduces the cost of implementing Early Stopping. We provide theoretical insights into the conditions under which this method is effective and empirically demonstrate its robust performance across standard benchmarks using common loss functions.