Neural Mutual Information Estimation with Vector Copulas

TL;DR

This paper introduces a new neural mutual information estimator based on vector copula theory that balances model complexity and capacity, improving estimation accuracy on diverse datasets.

Contribution

It proposes a novel interpolation method using vector copulas for mutual information estimation, bridging the gap between flexible neural models and simple Gaussian assumptions.

Findings

Outperforms existing estimators on synthetic benchmarks

Effective on real-world multimodal data

Balances complexity and capacity in MI estimation

Abstract

Estimating mutual information (MI) is a fundamental task in data science and machine learning. Existing estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, or overly simplified models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose a principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-art synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed estimator.