Deep Learning is Provably Robust to Symmetric Label Noise

TL;DR

This paper demonstrates that certain deep neural networks can be inherently robust to high levels of symmetric label noise, achieving optimal classification without mitigation strategies, under specific theoretical conditions.

Contribution

It provides a theoretical proof that $L_1$-consistent deep neural networks can attain Bayes optimality under symmetric label noise below a certain threshold, without needing noise mitigation.

Findings

DNNs can tolerate symmetric label noise up to the information-theoretic limit.

$L_1$-consistent DNN classifiers achieve Bayes optimality under symmetric noise.

No mitigation is necessary for $L_1$-consistent estimators under symmetric label noise.

Abstract

Deep neural networks (DNNs) are capable of perfectly fitting the training data, including memorizing noisy data. It is commonly believed that memorization hurts generalization. Therefore, many recent works propose mitigation strategies to avoid noisy data or correct memorization. In this work, we step back and ask the question: Can deep learning be robust against massive label noise without any mitigation? We provide an affirmative answer for the case of symmetric label noise: We find that certain DNNs, including under-parameterized and over-parameterized models, can tolerate massive symmetric label noise up to the information-theoretic threshold. By appealing to classical statistical theory and universal consistency of DNNs, we prove that for multiclass classification, $L_1$-consistent DNN classifiers trained under symmetric label noise can achieve Bayes optimality asymptotically if the label noise probability is less than $\frac{K-1}{K}$, where $K \ge 2$ is the number of classes. Our results show that for symmetric label noise, no mitigation is necessary for $L_1$-consistent estimators. We conjecture that for general label noise, mitigation strategies that make use of the noisy data will outperform those that ignore the noisy data.