Structured IB: Improving Information Bottleneck with Structured Feature Learning

TL;DR

Structured IB enhances the Information Bottleneck approach by integrating auxiliary encoders to extract structured features, leading to improved accuracy and information retention in neural networks.

Contribution

The paper introduces Structured IB, a novel framework that incorporates auxiliary encoders for extracting structured features, improving IB performance and robustness.

Findings

Superior prediction accuracy over traditional IB methods

Better preservation of task-relevant information

Effective with reduced network size

Abstract

The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size.