Distributed Information Bottleneck Method for Discrete and Gaussian Sources

TL;DR

This paper extends the information bottleneck method to distributed sources, providing theoretical characterizations and algorithms for both discrete and Gaussian models to optimize information preservation.

Contribution

It introduces a generalized distributed information bottleneck framework with single-letter characterizations and iterative algorithms for discrete and Gaussian sources.

Findings

Derived single-letter characterizations of the information-rate region.

Developed Blahut-Arimoto type algorithms for optimal trade-offs.

Applicable to both discrete memoryless and Gaussian sources.

Abstract

We study the problem of distributed information bottleneck, in which multiple encoders separately compress their observations in a manner such that, collectively, the compressed signals preserve as much information as possible about another signal. The model generalizes Tishby's centralized information bottleneck method to the setting of multiple distributed encoders. We establish single-letter characterizations of the information-rate region of this problem for both i) a class of discrete memoryless sources and ii) memoryless vector Gaussian sources. Furthermore, assuming a sum constraint on rate or complexity, for both models we develop Blahut-Arimoto type iterative algorithms that allow to compute optimal information-rate trade-offs, by iterating over a set of self-consistent equations.