A Sufficient-Statistic Reduction of the Information Bottleneck to a Low-Dimensional Problem

TL;DR

This paper demonstrates that the Information Bottleneck problem can be reduced to a low-dimensional form using sufficient statistics, making it computationally more tractable while preserving the full IB curve.

Contribution

It establishes an exact, loss-free reduction of the IB problem via sufficient statistics, linking classical Gaussian IB solutions to a broader nonlinear-Gaussian context.

Findings

Reduction preserves the IB curve and optimal representations.

Computational complexity depends on the sufficient statistic dimension.

Numerical example shows practical efficiency gains.

Abstract

We show that if the conditional distribution p(C | T) factors through a sufficient statistic {\phi}(T), then the Information Bottleneck (IB) problem for (T, C) is exactly equivalent to the IB problem for ({\phi}(T), C). The reduction is loss-free: it preserves the full IB curve, the Lagrangian optimum at every trade-off parameter \b{eta}, and the optimal representations up to pullback through {\phi}. As a result, the computational complexity of solving the IB problem is governed by the dimension of the sufficient statistic rather than the ambient dimension of the source. This identifies an exact structural condition under which the generic IB problem becomes tractable, and gives a formal bridge between the discrete and linear-Gaussian regimes. We then show that the classical Gaussian IB solution of Chechik, Globerson, Tishby and Weiss is an immediate corollary of this reduction, and we state a nonlinear-Gaussian generalisation. A small numerical example illustrates the practical consequence: when a low-dimensional sufficient statistic is available, the exact IB curve can be computed on the reduced problem at a cost determined by the statistic rather than by the ambient source dimension.