Toward Falsifying Causal Graphs Using a Permutation-Based Test

TL;DR

This paper introduces a permutation-based consistency metric for causal graphs that helps determine if a given graph significantly aligns with observed data, providing an interpretable baseline to assess correctness.

Contribution

It proposes a novel permutation-based baseline for evaluating the goodness of causal graphs, enabling significance testing against random permutations.

Findings

The metric effectively distinguishes true causal graphs from incorrect ones.

It performs well on both simulated and real-world datasets.

The approach offers an interpretable measure for causal graph validation.

Abstract

Understanding causal relationships among the variables of a system is paramount to explain and control its behavior. For many real-world systems, however, the true causal graph is not readily available and one must resort to predictions made by algorithms or domain experts. Therefore, metrics that quantitatively assess the goodness of a causal graph provide helpful checks before using it in downstream tasks. Existing metrics provide an $\textit{absolute}$ number of inconsistencies between the graph and the observed data, and without a baseline, practitioners are left to answer the hard question of how many such inconsistencies are acceptable or expected. Here, we propose a novel consistency metric by constructing a baseline through node permutations. By comparing the number of inconsistencies with those on the baseline, we derive an interpretable metric that captures whether the graph is significantly better than random. Evaluating on both simulated and real data sets from various domains, including biology and cloud monitoring, we demonstrate that the true graph is not falsified by our metric, whereas the wrong graphs given by a hypothetical user are likely to be falsified.