Scalable Mutual Information Estimation using Dependence Graphs

TL;DR

This paper introduces EDGE, a scalable non-parametric mutual information estimator that achieves optimal statistical and computational efficiency, enabling advanced analysis of deep neural networks.

Contribution

The paper presents EDGE, the first MI estimator with linear time complexity that attains parametric MSE rates, combining LSH, dependency graphs, and ensemble bias reduction.

Findings

EDGE achieves O(N) complexity and O(1/N) MSE rate.

EDGE enables analysis of information flow in deep neural networks.

The estimator clarifies the information bottleneck controversy in DNNs.

Abstract

The Mutual Information (MI) is an often used measure of dependency between two random variables utilized in information theory, statistics and machine learning. Recently several MI estimators have been proposed that can achieve parametric MSE convergence rate. However, most of the previously proposed estimators have the high computational complexity of at least $O(N^2)$. We propose a unified method for empirical non-parametric estimation of general MI function between random vectors in $\mathbb{R}^d$ based on $N$ i.i.d. samples. The reduced complexity MI estimator, called the ensemble dependency graph estimator (EDGE), combines randomized locality sensitive hashing (LSH), dependency graphs, and ensemble bias-reduction methods. We prove that EDGE achieves optimal computational complexity $O(N)$, and can achieve the optimal parametric MSE rate of $O(1/N)$ if the density is $d$ times differentiable. To the best of our knowledge EDGE is the first non-parametric MI estimator that can achieve parametric MSE rates with linear time complexity. We illustrate the utility of EDGE for the analysis of the information plane (IP) in deep learning. Using EDGE we shed light on a controversy on whether or not the compression property of information bottleneck (IB) in fact holds for ReLu and other rectification functions in deep neural networks (DNN).