Bottleneck Problems: Information and Estimation-Theoretic View

TL;DR

This paper unifies the theoretical understanding of information bottleneck and privacy funnel, introduces a broader class of bottleneck problems with alternative mutual information measures, and provides practical bounds and closed-form solutions for these complex optimization problems.

Contribution

It offers a unified theoretical framework for IB and PF, introduces generalized bottleneck problems with alternative mutual information, and develops methods for their tractable evaluation and closed-form solutions.

Findings

Proved a new cardinality bound for the auxiliary variable in IB.

Established connections between IB/PF and three information-theoretic coding problems.

Derived closed-form solutions for several bottleneck problems in the binary case.

Abstract

Information bottleneck (IB) and privacy funnel (PF) are two closely related optimization problems which have found applications in machine learning, design of privacy algorithms, capacity problems (e.g., Mrs. Gerber's Lemma), strong data processing inequalities, among others. In this work, we first investigate the functional properties of IB and PF through a unified theoretical framework. We then connect them to three information-theoretic coding problems, namely hypothesis testing against independence, noisy source coding and dependence dilution. Leveraging these connections, we prove a new cardinality bound for the auxiliary variable in IB, making its computation more tractable for discrete random variables. In the second part, we introduce a general family of optimization problems, termed as \textit{bottleneck problems}, by replacing mutual information in IB and PF with other notions of mutual information, namely $f$-information and Arimoto's mutual information. We then argue that, unlike IB and PF, these problems lead to easily interpretable guarantee in a variety of inference tasks with statistical constraints on accuracy and privacy. Although the underlying optimization problems are non-convex, we develop a technique to evaluate bottleneck problems in closed form by equivalently expressing them in terms of lower convex or upper concave envelope of certain functions. By applying this technique to binary case, we derive closed form expressions for several bottleneck problems.