Potential Conditional Mutual Information: Estimators, Properties and Applications

TL;DR

This paper introduces potential conditional mutual information, a new functional based on a modified distribution, along with KNN estimators, and demonstrates its theoretical properties and practical utility in dynamical systems.

Contribution

The paper defines potential conditional mutual information, develops KNN estimators with importance sampling, and proves their consistency, advancing tools for conditional independence and causal inference.

Findings

Estimator shows excellent practical performance

Proven finite k consistency of the estimator

Application demonstrated in dynamical system inference

Abstract

The conditional mutual information I(X;Y|Z) measures the average information that X and Y contain about each other given Z. This is an important primitive in many learning problems including conditional independence testing, graphical model inference, causal strength estimation and time-series problems. In several applications, it is desirable to have a functional purely of the conditional distribution p_{Y|X,Z} rather than of the joint distribution p_{X,Y,Z}. We define the potential conditional mutual information as the conditional mutual information calculated with a modified joint distribution p_{Y|X,Z} q_{X,Z}, where q_{X,Z} is a potential distribution, fixed airport. We develop K nearest neighbor based estimators for this functional, employing importance sampling, and a coupling trick, and prove the finite k consistency of such an estimator. We demonstrate that the estimator has excellent practical performance and show an application in dynamical system inference.