CLID-MU: Cross-Layer Information Divergence Based Meta Update Strategy for Learning with Noisy Labels

TL;DR

This paper introduces CLID-MU, a meta-update strategy that uses cross-layer information divergence to improve learning with noisy labels without requiring a clean meta-dataset, showing superior results on benchmarks.

Contribution

Proposes a novel meta-learning method that leverages data structure consistency across neural network layers to handle noisy labels without clean meta-data.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Effectively handles both synthetic and real-world label noise.

Does not rely on a clean labeled meta-dataset.

Abstract

Learning with noisy labels (LNL) is essential for training deep neural networks with imperfect data. Meta-learning approaches have achieved success by using a clean unbiased labeled set to train a robust model. However, this approach heavily depends on the availability of a clean labeled meta-dataset, which is difficult to obtain in practice. In this work, we thus tackle the challenge of meta-learning for noisy label scenarios without relying on a clean labeled dataset. Our approach leverages the data itself while bypassing the need for labels. Building on the insight that clean samples effectively preserve the consistency of related data structures across the last hidden and the final layer, whereas noisy samples disrupt this consistency, we design the Cross-layer Information Divergence-based Meta Update Strategy (CLID-MU). CLID-MU leverages the alignment of data structures across these diverse feature spaces to evaluate model performance and use this alignment to guide training. Experiments on benchmark datasets with varying amounts of labels under both synthetic and real-world noise demonstrate that CLID-MU outperforms state-of-the-art methods. The code is released at https://github.com/ruofanhu/CLID-MU.