On Robust Learning from Noisy Labels: A Permutation Layer Approach

TL;DR

This paper introduces PermLL, a permutation layer method that dynamically calibrates deep neural networks to better handle label noise, improving robustness and achieving state-of-the-art results.

Contribution

It proposes a novel permutation layer approach for robust learning from noisy labels, with theoretical analysis and two variants enhancing existing methods.

Findings

PermLL outperforms existing methods on real and synthetic datasets.

The permutation layer effectively mitigates label noise impact.

Theoretical analysis clarifies the relationship between variants.

Abstract

The existence of label noise imposes significant challenges (e.g., poor generalization) on the training process of deep neural networks (DNN). As a remedy, this paper introduces a permutation layer learning approach termed PermLL to dynamically calibrate the training process of the DNN subject to instance-dependent and instance-independent label noise. The proposed method augments the architecture of a conventional DNN by an instance-dependent permutation layer. This layer is essentially a convex combination of permutation matrices that is dynamically calibrated for each sample. The primary objective of the permutation layer is to correct the loss of noisy samples mitigating the effect of label noise. We provide two variants of PermLL in this paper: one applies the permutation layer to the model's prediction, while the other applies it directly to the given noisy label. In addition, we provide a theoretical comparison between the two variants and show that previous methods can be seen as one of the variants. Finally, we validate PermLL experimentally and show that it achieves state-of-the-art performance on both real and synthetic datasets.