Stable and Convexified Information Bottleneck Optimization via Symbolic Continuation and Entropy-Regularized Trajectories

TL;DR

This paper introduces a stable, convexified approach to the Information Bottleneck method using symbolic continuation and entropy regularization, ensuring robust optimization and consistent representations.

Contribution

It analytically proves convexity and uniqueness of IB solutions with entropy regularization, enhancing stability and providing a practical framework for implementation.

Findings

Proves convexity and uniqueness of IB solutions with entropy regularization.

Demonstrates stabilized representation learning across various beta values.

Provides extensive sensitivity analysis with uncertainty quantification.

Abstract

The Information Bottleneck (IB) method frequently suffers from unstable optimization, characterized by abrupt representation shifts near critical points of the IB trade-off parameter, beta. In this paper, I introduce a novel approach to achieve stable and convex IB optimization through symbolic continuation and entropy-regularized trajectories. I analytically prove convexity and uniqueness of the IB solution path when an entropy regularization term is included, and demonstrate how this stabilizes representation learning across a wide range of \b{eta} values. Additionally, I provide extensive sensitivity analyses around critical points (beta) with statistically robust uncertainty quantification (95% confidence intervals). The open-source implementation, experimental results, and reproducibility framework included in this work offer a clear path for practical deployment and future extension of my proposed method.