A Generalized Information Bottleneck Method: A Decision-Theoretic Perspective

TL;DR

This paper introduces a generalized information bottleneck framework using a decision-theoretic perspective, enabling more flexible tradeoffs between data compression and relevance preservation.

Contribution

It extends the classical IB method by incorporating a broader class of information measures with a decision-theoretic interpretation and proposes an optimization algorithm for this generalized setting.

Findings

The generalized IB framework can evaluate utility using $\\mathcal{H}$-mutual information.

An alternating optimization algorithm effectively balances compression and utility.

The approach provides a new perspective on information bottleneck tradeoffs.

Abstract

The information bottleneck (IB) method seeks a compressed representation of data that preserves information relevant to a target variable for prediction while discarding irrelevant information from the original data. In its classical formulation, the IB method employs mutual information to evaluate the compression between the original and compressed data and the utility of the representation for the target variable. In this study, we investigate a generalized IB problem, where the evaluation of utility is based on the $\mathcal{H}$-mutual information that satisfies the concave (\texttt{CV}) and averaging (\texttt{AVG}) conditions. This class of information measures admits a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information. Based on this interpretation, we derive an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem.