Diffeomorphic Information Neural Estimation

TL;DR

This paper introduces DINE, a novel neural estimator for conditional mutual information that efficiently handles complex, high-dimensional data by leveraging diffeomorphic invariance, outperforming existing methods in various tasks.

Contribution

DINE is a new approach that estimates CMI by using diffeomorphic invariance, enabling efficient and accurate estimation in complex, high-dimensional scenarios.

Findings

DINE outperforms state-of-the-art estimators in MI and CMI tasks.

DINE effectively handles high-dimensional and complex relationships.

DINE improves conditional independence testing accuracy.

Abstract

Mutual Information (MI) and Conditional Mutual Information (CMI) are multi-purpose tools from information theory that are able to naturally measure the statistical dependencies between random variables, thus they are usually of central interest in several statistical and machine learning tasks, such as conditional independence testing and representation learning. However, estimating CMI, or even MI, is infamously challenging due the intractable formulation. In this study, we introduce DINE (Diffeomorphic Information Neural Estimator)-a novel approach for estimating CMI of continuous random variables, inspired by the invariance of CMI over diffeomorphic maps. We show that the variables of interest can be replaced with appropriate surrogates that follow simpler distributions, allowing the CMI to be efficiently evaluated via analytical solutions. Additionally, we demonstrate the quality of the proposed estimator in comparison with state-of-the-arts in three important tasks, including estimating MI, CMI, as well as its application in conditional independence testing. The empirical evaluations show that DINE consistently outperforms competitors in all tasks and is able to adapt very well to complex and high-dimensional relationships.