Noisy Concurrent Training for Efficient Learning under Label Noise

TL;DR

This paper introduces Noisy Concurrent Training (NCT), a collaborative learning method with label variability regularization, to improve deep neural network training under label noise by reducing memorization and enhancing generalization.

Contribution

The paper proposes NCT, a novel training framework combining collaborative learning and target variability regularization to combat label noise in deep neural networks.

Findings

NCT improves accuracy on noisy datasets.

Target variability reduces overfitting to noisy labels.

Models rely more on consensus as training progresses.

Abstract

Deep neural networks (DNNs) fail to learn effectively under label noise and have been shown to memorize random labels which affect their generalization performance. We consider learning in isolation, using one-hot encoded labels as the sole source of supervision, and a lack of regularization to discourage memorization as the major shortcomings of the standard training procedure. Thus, we propose Noisy Concurrent Training (NCT) which leverages collaborative learning to use the consensus between two models as an additional source of supervision. Furthermore, inspired by trial-to-trial variability in the brain, we propose a counter-intuitive regularization technique, target variability, which entails randomly changing the labels of a percentage of training samples in each batch as a deterrent to memorization and over-generalization in DNNs. Target variability is applied independently to each model to keep them diverged and avoid the confirmation bias. As DNNs tend to prioritize learning simple patterns first before memorizing the noisy labels, we employ a dynamic learning scheme whereby as the training progresses, the two models increasingly rely more on their consensus. NCT also progressively increases the target variability to avoid memorization in later stages. We demonstrate the effectiveness of our approach on both synthetic and real-world noisy benchmark datasets.