Information-Theoretic Approximation to Causal Models

TL;DR

This paper introduces an information-theoretic method for inferring causal directions between discrete variables by embedding distributions into a higher-dimensional space and approximating them with causal models, enabling causal discovery and probability estimation.

Contribution

It proposes a novel approximation method (IACM) using relative entropy minimization to infer causal models from empirical data, including causal discovery and probability calculations.

Findings

Method can approximate causal models via linear optimization.

Approach estimates probabilities of causation in causal models.

Performance varies; competitive on some real-world and synthetic data.

Abstract

Inferring the causal direction and causal effect between two discrete random variables X and Y from a finite sample is often a crucial problem and a challenging task. However, if we have access to observational and interventional data, it is possible to solve that task. If X is causing Y, then it does not matter if we observe an effect in Y by observing changes in X or by intervening actively on X. This invariance principle creates a link between observational and interventional distributions in a higher dimensional probability space. We embed distributions that originate from samples of X and Y into that higher dimensional space such that the embedded distribution is closest to the distributions that follow the invariance principle, with respect to the relative entropy. This allows us to calculate the best information-theoretic approximation for a given empirical distribution, that follows an assumed underlying causal model. We show that this information-theoretic approximation to causal models (IACM) can be done by solving a linear optimization problem. In particular, by approximating the empirical distribution to a monotonic causal model, we can calculate probabilities of causation. We can also use IACM for causal discovery problems in the bivariate, discrete case. However, experimental results on labeled synthetic data from additive noise models show that our causal discovery approach is lagging behind state-of-the-art approaches because the invariance principle encodes only a necessary condition for causal relations. Nevertheless, for synthetic multiplicative noise data and real-world data, our approach can compete in some cases with alternative methods.