A Fast Kernel-based Conditional Independence test with Application to Causal Discovery

TL;DR

FastKCI is a scalable, parallel kernel-based conditional independence test that significantly reduces computation time while maintaining statistical power, enabling causal discovery in large datasets.

Contribution

We introduce FastKCI, a novel parallelizable method that accelerates kernel-based conditional independence testing using dataset partitioning and importance sampling.

Findings

FastKCI achieves substantial speedups over traditional KCI tests.

It maintains comparable statistical power to the original KCI.

Validated on synthetic and real-world datasets.

Abstract

Kernel-based conditional independence (KCI) testing is a powerful nonparametric method commonly employed in causal discovery tasks. Despite its flexibility and statistical reliability, cubic computational complexity limits its application to large datasets. To address this computational bottleneck, we propose \textit{FastKCI}, a scalable and parallelizable kernel-based conditional independence test that utilizes a mixture-of-experts approach inspired by embarrassingly parallel inference techniques for Gaussian processes. By partitioning the dataset based on a Gaussian mixture model over the conditioning variables, FastKCI conducts local KCI tests in parallel, aggregating the results using an importance-weighted sampling scheme. Experiments on synthetic datasets and benchmarks on real-world production data validate that FastKCI maintains the statistical power of the original KCI test while achieving substantial computational speedups. FastKCI thus represents a practical and efficient solution for conditional independence testing in causal inference on large-scale data.