Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

TL;DR

This paper introduces an entropy-regularized optimal transport approach to the information bottleneck problem, enabling more flexible solutions beyond the limitations of traditional algorithms, with demonstrated efficiency and effectiveness.

Contribution

It proposes a novel OT-based framework for IB from a posterior perspective, generalizing the Sinkhorn algorithm for improved optimization in machine learning.

Findings

OT model effectively solves IB problem

Generalized Sinkhorn algorithm improves computational efficiency

Numerical experiments validate approach's effectiveness

Abstract

Information bottleneck (IB) is a paradigm to extract information in one target random variable from another relevant random variable, which has aroused great interest due to its potential to explain deep neural networks in terms of information compression and prediction. Despite its great importance, finding the optimal bottleneck variable involves a difficult nonconvex optimization problem due to the nonconvexity of mutual information constraint. The Blahut-Arimoto algorithm and its variants provide an approach by considering its Lagrangian with fixed Lagrange multiplier. However, only the strictly concave IB curve can be fully obtained by the BA algorithm, which strongly limits its application in machine learning and related fields, as strict concavity cannot be guaranteed in those problems. To overcome the above difficulty, we derive an entropy regularized optimal transport (OT) model for IB problem from a posterior probability perspective. Correspondingly, we use the alternating optimization procedure and generalize the Sinkhorn algorithm to solve the above OT model. The effectiveness and efficiency of our approach are demonstrated via numerical experiments.