Max-Sliced Mutual Information

TL;DR

This paper introduces max-sliced mutual information (mSMI), a scalable and efficient dependence measure that captures complex relationships in high-dimensional data, bridging the gap between CCA and mutual information.

Contribution

It proposes mSMI, a novel information-theoretic measure that generalizes CCA to high dimensions, with a neural estimator and theoretical guarantees, applicable to various learning tasks.

Findings

mSMI effectively captures intricate dependencies in high-dimensional data.

The neural estimator of mSMI provides fast and scalable estimation with error bounds.

Experiments show mSMI outperforms existing methods in multiple applications.

Abstract

Quantifying the dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure that also captures high-order dependencies. However, CCA only accounts for linear dependence, which may be insufficient for certain applications, while mutual information is often infeasible to compute/estimate in high dimensions. This work proposes a middle ground in the form of a scalable information-theoretic generalization of CCA, termed max-sliced mutual information (mSMI). mSMI equals the maximal mutual information between low-dimensional projections of the high-dimensional variables, which reduces back to CCA in the Gaussian case. It enjoys the best of both worlds: capturing intricate dependencies in the data while being amenable to fast computation and scalable estimation from samples. We show that mSMI retains favorable structural properties of Shannon's mutual information, like variational forms and identification of independence. We then study statistical estimation of mSMI, propose an efficiently computable neural estimator, and couple it with formal non-asymptotic error bounds. We present experiments that demonstrate the utility of mSMI for several tasks, encompassing independence testing, multi-view representation learning, algorithmic fairness, and generative modeling. We observe that mSMI consistently outperforms competing methods with little-to-no computational overhead.