On the Hardness of Conditional Independence Testing In Practice

TL;DR

This paper investigates the practical challenges of conditional independence testing, revealing how errors in kernel estimates and kernel choices impact test validity and power in real-world applications.

Contribution

It identifies key factors affecting the performance of kernel-based CI tests, especially the role of conditional mean embedding errors and kernel selection.

Findings

Errors in conditional mean embedding affect Type-I error.

Choosing the right conditioning kernel is crucial for test power.

Kernel-based CI tests often face practical limitations due to these factors.

Abstract

Tests of conditional independence (CI) underpin a number of important problems in machine learning and statistics, from causal discovery to evaluation of predictor fairness and out-of-distribution robustness. Shah and Peters (2020) showed that, contrary to the unconditional case, no universally finite-sample valid test can ever achieve nontrivial power. While informative, this result (based on "hiding" dependence) does not seem to explain the frequent practical failures observed with popular CI tests. We investigate the Kernel-based Conditional Independence (KCI) test - of which we show the Generalized Covariance Measure underlying many recent tests is nearly a special case - and identify the major factors underlying its practical behavior. We highlight the key role of errors in the conditional mean embedding estimate for the Type-I error, while pointing out the importance of selecting an appropriate conditioning kernel (not recognized in previous work) as being necessary for good test power but also tending to inflate Type-I error.