Towards Robust Learning with Different Label Noise Distributions

TL;DR

This paper introduces a method for robust learning with label noise by relabeling images to handle different noise distributions and applying semi-supervised learning twice, improving performance on noisy datasets.

Contribution

It proposes a novel relabeling strategy and a double SSL approach to better detect and learn from clean samples under various label noise distributions.

Findings

Non-uniform out-of-distribution noise resembles real-world noise.

Intermediate features are often unaffected by label noise.

The method outperforms recent state-of-the-art on multiple datasets.

Abstract

Noisy labels are an unavoidable consequence of labeling processes and detecting them is an important step towards preventing performance degradations in Convolutional Neural Networks. Discarding noisy labels avoids a harmful memorization, while the associated image content can still be exploited in a semi-supervised learning (SSL) setup. Clean samples are usually identified using the small loss trick, i.e. they exhibit a low loss. However, we show that different noise distributions make the application of this trick less straightforward and propose to continuously relabel all images to reveal a discriminative loss against multiple distributions. SSL is then applied twice, once to improve the clean-noisy detection and again for training the final model. We design an experimental setup based on ImageNet32/64 for better understanding the consequences of representation learning with differing label noise distributions and find that non-uniform out-of-distribution noise better resembles real-world noise and that in most cases intermediate features are not affected by label noise corruption. Experiments in CIFAR-10/100, ImageNet32/64 and WebVision (real-world noise) demonstrate that the proposed label noise Distribution Robust Pseudo-Labeling (DRPL) approach gives substantial improvements over recent state-of-the-art. Code is available at https://git.io/JJ0PV.