Identifying Training Stop Point with Noisy Labeled Data

TL;DR

This paper introduces AutoTSP, a novel method that automatically determines the optimal training stopping point for deep neural networks trained on noisy data, solely based on training behavior without requiring clean validation data or noise estimation.

Contribution

The paper proposes a training stop point detection method that relies only on training accuracy trends, eliminating the need for clean validation sets or noise ratio knowledge.

Findings

AutoTSP effectively finds near-optimal stopping points across various datasets and noise conditions.

The method is robust to different noise ratios and types, maintaining high test accuracy.

AutoTSP outperforms existing early stopping techniques that require additional data or noise estimates.

Abstract

Training deep neural networks (DNNs) with noisy labels is a challenging problem due to over-parameterization. DNNs tend to essentially fit on clean samples at a higher rate in the initial stages, and later fit on the noisy samples at a relatively lower rate. Thus, with a noisy dataset, the test accuracy increases initially and drops in the later stages. To find an early stopping point at the maximum obtainable test accuracy (MOTA), recent studies assume either that i) a clean validation set is available or ii) the noise ratio is known, or, both. However, often a clean validation set is unavailable, and the noise estimation can be inaccurate. To overcome these issues, we provide a novel training solution, free of these conditions. We analyze the rate of change of the training accuracy for different noise ratios under different conditions to identify a training stop region. We further develop a heuristic algorithm based on a small-learning assumption to find a training stop point (TSP) at or close to MOTA. To the best of our knowledge, our method is the first to rely solely on the \textit{training behavior}, while utilizing the entire training set, to automatically find a TSP. We validated the robustness of our algorithm (AutoTSP) through several experiments on CIFAR-10, CIFAR-100, and a real-world noisy dataset for different noise ratios, noise types, and architectures.