Distilling Effective Supervision from Severe Label Noise

TL;DR

This paper introduces a robust training framework that effectively handles high levels of label noise by leveraging a small trusted set to estimate exemplar weights and pseudo labels, significantly improving accuracy on noisy datasets.

Contribution

The paper proposes a holistic method that uses a small trusted set to estimate exemplar weights and pseudo labels, enhancing neural network robustness against severe label noise.

Findings

Achieves 80.2% accuracy on CIFAR100 with 40% uniform noise and minimal trusted data.

Maintains 75.5% accuracy with 80% noise, outperforming previous methods.

Sets new state-of-the-art performance on various noisy label benchmarks.

Abstract

Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a $40\%$ uniform noise ratio and only 10 trusted labeled data per class, our method achieves $80.2{\pm}0.3\%$ classification accuracy, where the error rate is only $1.4\%$ higher than a neural network trained without label noise. Moreover, increasing the noise ratio to $80\%$, our method still maintains a high accuracy of $75.5{\pm}0.2\%$, compared to the previous best accuracy $48.2\%$. Source code available: https://github.com/google-research/google-research/tree/master/ieg