High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction

TL;DR

This paper introduces a PCA-based method for differential entropy estimation in high dimensions, reducing sample complexity and enabling analysis of neural network information flow.

Contribution

The paper proposes a novel PCA-based approach to estimate differential entropy in high dimensions, overcoming exponential sample complexity issues.

Findings

PCA projection reduces sample complexity in entropy estimation.

Method achieves near-optimal performance for low-dimensional structures.

Applied to neural networks, it measures mutual information effectively.

Abstract

We study the problem of overcoming exponential sample complexity in differential entropy estimation under Gaussian convolutions. Specifically, we consider the estimation of the differential entropy $h(X+Z)$ via $n$ independently and identically distributed samples of $X$, where $X$ and $Z$ are independent $D$-dimensional random variables with $X$ sub-Gaussian with bounded second moment and $Z\sim\mathcal{N}(0,\sigma^2I_D)$. Under the absolute-error loss, the above problem has a parametric estimation rate of $\frac{c^D}{\sqrt{n}}$, which is exponential in data dimension $D$ and often problematic for applications. We overcome this exponential sample complexity by projecting $X$ to a low-dimensional space via principal component analysis (PCA) before the entropy estimation, and show that the asymptotic error overhead vanishes as the unexplained variance of the PCA vanishes. This implies near-optimal performance for inherently low-dimensional structures embedded in high-dimensional spaces, including hidden-layer outputs of deep neural networks (DNN), which can be used to estimate mutual information (MI) in DNNs. We provide numerical results verifying the performance of our PCA approach on Gaussian and spiral data. We also apply our method to analysis of information flow through neural network layers (c.f. information bottleneck), with results measuring mutual information in a noisy fully connected network and a noisy convolutional neural network (CNN) for MNIST classification.