An Achievable and Analytic Solution to Information Bottleneck for Gaussian Mixtures

TL;DR

This paper presents a novel, closed-form solution for the information bottleneck problem involving Gaussian mixtures, offering near-optimal compression strategies with applications in machine learning and image classification.

Contribution

It introduces a unified achievable scheme with three compression strategies for Gaussian mixture IB problems, extending analysis to vector cases and real-world datasets.

Findings

Near-optimal performance across various SNRs

Outperforms Blahut-Arimoto algorithm and information dropout

Improves classification accuracy on MNIST dataset

Abstract

In this paper, we study a remote source coding scenario in which binary phase shift keying (BPSK) modulation sources are corrupted by additive white Gaussian noise (AWGN). An intermediate node, such as a relay, receives these observations and performs additional compression to balance complexity and relevance. This problem can be further formulated as an information bottleneck (IB) problem with Bernoulli sources and Gaussian mixture observations. However, no closed-form solution exists for this IB problem. To address this challenge, we propose a unified achievable scheme that employs three different compression/quantization strategies for intermediate node processing by using two-level quantization, multi-level deterministic quantization, and soft quantization with the hyperbolic tangent ($\tanh$) function, respectively. In addition, we extend our analysis to the vector mixture Gaussian observation problem and explore its application in machine learning for binary classification with information leakage. Numerical evaluations show that the proposed scheme has a near-optimal performance over various signal-to-noise ratios (SNRs), compared to the Blahut-Arimoto (BA) algorithm, and has better performance than some existing numerical methods such as the information dropout approach. Furthermore, experiments conducted on the realistic MNIST dataset also validate the superior classification accuracy of our method compared to the information dropout approach.