Multivariate Information Bottleneck

TL;DR

This paper extends the information bottleneck method to multivariate data using Bayesian networks, enabling analysis of interrelated data partitions and providing new insights and algorithms for complex data clustering.

Contribution

It introduces a principled multivariate framework for the information bottleneck using Bayesian networks, allowing for interconnected data partitions and iterative solution algorithms.

Findings

Framework for multivariate information bottleneck using Bayesian networks

Characterization of solution variations and insights into bottleneck behavior

Development of iterative algorithms for constructing solutions

Abstract

The Information bottleneck method is an unsupervised non-parametric data organization technique. Given a joint distribution P(A,B), this method constructs a new variable T that extracts partitions, or clusters, over the values of A that are informative about B. The information bottleneck has already been applied to document classification, gene expression, neural code, and spectral analysis. In this paper, we introduce a general principled framework for multivariate extensions of the information bottleneck method. This allows us to consider multiple systems of data partitions that are inter-related. Our approach utilizes Bayesian networks for specifying the systems of clusters and what information each captures. We show that this construction provides insight about bottleneck variations and enables us to characterize solutions of these variations. We also present a general framework for iterative algorithms for constructing solutions, and apply it to several examples.