A Neural Difference-of-Entropies Estimator for Mutual Information

TL;DR

This paper introduces a new neural estimator for mutual information using normalizing flows, which improves bias-variance trade-offs in high-dimensional dependence estimation tasks.

Contribution

It proposes a novel MI estimator based on normalizing flows with a block autoregressive structure, enhancing estimation accuracy in high dimensions.

Findings

Improved bias-variance trade-off on benchmark tasks

Effective estimation of mutual information in high-dimensional settings

Utilizes normalizing flows for flexible density modeling

Abstract

Estimating Mutual Information (MI), a key measure of dependence of random quantities without specific modelling assumptions, is a challenging problem in high dimensions. We propose a novel mutual information estimator based on parametrizing conditional densities using normalizing flows, a deep generative model that has gained popularity in recent years. This estimator leverages a block autoregressive structure to achieve improved bias-variance trade-offs on standard benchmark tasks.