Statistically Valid Information Bottleneck via Multiple Hypothesis Testing

TL;DR

This paper introduces IB-MHT, a statistically valid method for solving the information bottleneck problem that guarantees IB constraints with high probability, improving robustness over heuristic approaches.

Contribution

It presents a novel IB solution using multiple hypothesis testing that provides statistical guarantees, applicable to classical and deterministic IB problems.

Findings

IB-MHT outperforms traditional methods in robustness

Provides high-probability guarantees on IB constraints

Effective in language model distillation

Abstract

The information bottleneck (IB) problem is a widely studied framework in machine learning for extracting compressed features that are informative for downstream tasks. However, current approaches to solving the IB problem rely on a heuristic tuning of hyperparameters, offering no guarantees that the learned features satisfy information-theoretic constraints. In this work, we introduce a statistically valid solution to this problem, referred to as IB via multiple hypothesis testing (IB-MHT), which ensures that the learned features meet the IB constraints with high probability, regardless of the size of the available dataset. The proposed methodology builds on Pareto testing and learn-then-test (LTT), and it wraps around existing IB solvers to provide statistical guarantees on the IB constraints. We demonstrate the performance of IB-MHT on classical and deterministic IB formulations, including experiments on distillation of language models. The results validate the effectiveness of IB-MHT in outperforming conventional methods in terms of statistical robustness and reliability.