Learning with Instance-Dependent Label Noise: A Sample Sieve Approach

TL;DR

This paper introduces CORES$^{2}$, a theoretically grounded sample sieve method that effectively filters out instance-dependent label noise in training deep neural networks, improving robustness without needing prior noise rate estimation.

Contribution

Proposes CORES$^{2}$, a noise filtering technique that does not require noise rate estimation and provides theoretical guarantees for handling instance-dependent label noise.

Findings

CORES$^{2}$ effectively filters noisy labels on CIFAR datasets.

The method improves deep neural network performance under real-world noisy labels.

Provides a generic framework for analyzing and improving noisy datasets.

Abstract

Human-annotated labels are often prone to noise, and the presence of such noise will degrade the performance of the resulting deep neural network (DNN) models. Much of the literature (with several recent exceptions) of learning with noisy labels focuses on the case when the label noise is independent of features. Practically, annotations errors tend to be instance-dependent and often depend on the difficulty levels of recognizing a certain task. Applying existing results from instance-independent settings would require a significant amount of estimation of noise rates. Therefore, providing theoretically rigorous solutions for learning with instance-dependent label noise remains a challenge. In this paper, we propose CORES$^{2}$ (COnfidence REgularized Sample Sieve), which progressively sieves out corrupted examples. The implementation of CORES$^{2}$ does not require specifying noise rates and yet we are able to provide theoretical guarantees of CORES$^{2}$ in filtering out the corrupted examples. This high-quality sample sieve allows us to treat clean examples and the corrupted ones separately in training a DNN solution, and such a separation is shown to be advantageous in the instance-dependent noise setting. We demonstrate the performance of CORES$^{2}$ on CIFAR10 and CIFAR100 datasets with synthetic instance-dependent label noise and Clothing1M with real-world human noise. As of independent interests, our sample sieve provides a generic machinery for anatomizing noisy datasets and provides a flexible interface for various robust training techniques to further improve the performance. Code is available at https://github.com/UCSC-REAL/cores.