Is the Information Bottleneck Robust Enough? Towards Label-Noise Resistant Information Bottleneck Learning

TL;DR

This paper introduces LaT-IB, a novel information bottleneck method that enhances robustness to label noise by disentangling clean and noisy label information, supported by theoretical bounds and a three-phase training framework.

Contribution

LaT-IB is the first IB-based approach with a mutual information regularizer and noise-aware disentanglement designed specifically for label-noise resistance.

Findings

LaT-IB outperforms existing methods under label noise.

Theoretical bounds support noise invariance and separation.

Three-phase training improves robustness and efficiency.

Abstract

The Information Bottleneck (IB) principle facilitates effective representation learning by preserving label-relevant information while compressing irrelevant information. However, its strong reliance on accurate labels makes it inherently vulnerable to label noise, prevalent in real-world scenarios, resulting in significant performance degradation and overfitting. To address this issue, we propose LaT-IB, a novel Label-Noise ResistanT Information Bottleneck method which introduces a "Minimal-Sufficient-Clean" (MSC) criterion. Instantiated as a mutual information regularizer to retain task-relevant information while discarding noise, MSC addresses standard IB's vulnerability to noisy label supervision. To achieve this, LaT-IB employs a noise-aware latent disentanglement that decomposes the latent representation into components aligned with to the clean label space and the noise space. Theoretically, we first derive mutual information bounds for each component of our objective including prediction, compression, and disentanglement, and moreover prove that optimizing it encourages representations invariant to input noise and separates clean and noisy label information. Furthermore, we design a three-phase training framework: Warmup, Knowledge Injection and Robust Training, to progressively guide the model toward noise-resistant representations. Extensive experiments demonstrate that LaT-IB achieves superior robustness and efficiency under label noise, significantly enhancing robustness and applicability in real-world scenarios with label noise.