Late Stopping: Avoiding Confidently Learning from Mislabeled Examples

TL;DR

This paper introduces Late Stopping, a training framework that improves learning from noisy labels by gradually removing likely mislabeled examples during prolonged training, thereby enhancing model robustness.

Contribution

The paper proposes a novel Late Stopping framework that leverages the intrinsic robustness of DNNs to better identify and remove mislabeled data over extended training.

Findings

Late Stopping effectively removes high-probability mislabeled examples.

The method outperforms existing approaches on benchmark noisy datasets.

Clean hard examples are retained throughout the training process.

Abstract

Sample selection is a prevalent method in learning with noisy labels, where small-loss data are typically considered as correctly labeled data. However, this method may not effectively identify clean hard examples with large losses, which are critical for achieving the model's close-to-optimal generalization performance. In this paper, we propose a new framework, Late Stopping, which leverages the intrinsic robust learning ability of DNNs through a prolonged training process. Specifically, Late Stopping gradually shrinks the noisy dataset by removing high-probability mislabeled examples while retaining the majority of clean hard examples in the training set throughout the learning process. We empirically observe that mislabeled and clean examples exhibit differences in the number of epochs required for them to be consistently and correctly classified, and thus high-probability mislabeled examples can be removed. Experimental results on benchmark-simulated and real-world noisy datasets demonstrate that the proposed method outperforms state-of-the-art counterparts.