Learning from Noisy Labels via Dynamic Loss Thresholding

TL;DR

This paper introduces Dynamic Loss Thresholding (DLT), a method that improves learning from noisy labels by dynamically filtering potentially corrupted data during training, leading to better generalization in deep neural networks.

Contribution

The paper proposes DLT, a novel dynamic thresholding approach that effectively identifies clean samples in noisy datasets and introduces methods to estimate noise rates and analyze hard samples.

Findings

DLT outperforms state-of-the-art methods on CIFAR-10/100 and Clothing1M.

Proposes a new way to estimate dataset noise rates based on loss differences.

Investigates the impact of hard samples on learning from noisy labels.

Abstract

Numerous researches have proved that deep neural networks (DNNs) can fit everything in the end even given data with noisy labels, and result in poor generalization performance. However, recent studies suggest that DNNs tend to gradually memorize the data, moving from correct data to mislabeled data. Inspired by this finding, we propose a novel method named Dynamic Loss Thresholding (DLT). During the training process, DLT records the loss value of each sample and calculates dynamic loss thresholds. Specifically, DLT compares the loss value of each sample with the current loss threshold. Samples with smaller losses can be considered as clean samples with higher probability and vice versa. Then, DLT discards the potentially corrupted labels and further leverages supervised learning techniques. Experiments on CIFAR-10/100 and Clothing1M demonstrate substantial improvements over recent state-of-the-art methods. In addition, we investigate two real-world problems for the first time. Firstly, we propose a novel approach to estimate the noise rates of datasets based on the loss difference between the early and late training stages of DNNs. Secondly, we explore the effect of hard samples (which are difficult to be distinguished) on the process of learning from noisy labels.