On the Relevance-Complexity Region of Scalable Information Bottleneck

TL;DR

This paper explores the scalable information bottleneck, characterizing the tradeoff between relevance and complexity for different sources, and provides algorithms and examples demonstrating its application in pattern classification.

Contribution

It introduces the relevance-complexity region for scalable information bottleneck and develops algorithms for its computation, extending the understanding of this method.

Findings

Explicit relevance-complexity regions for Gaussian and binary sources.

A Blahut-Arimoto type algorithm for general discrete sources.

Numerical results demonstrating application in pattern classification.

Abstract

The Information Bottleneck method is a learning technique that seeks a right balance between accuracy and generalization capability through a suitable tradeoff between compression complexity, measured by minimum description length, and distortion evaluated under logarithmic loss measure. In this paper, we study a variation of the problem, called scalable information bottleneck, where the encoder outputs multiple descriptions of the observation with increasingly richer features. The problem at hand is motivated by some application scenarios that require varying levels of accuracy depending on the allowed level of generalization. First, we establish explicit (analytic) characterizations of the relevance-complexity region for memoryless Gaussian sources and memoryless binary sources. Then, we derive a Blahut-Arimoto type algorithm that allows us to compute (an approximation of) the region for general discrete sources. Finally, an application example in the pattern classification problem is provided along with numerical results.