Causal KL: Evaluating Causal Discovery

TL;DR

This paper introduces Causal KL, an augmented divergence metric that better evaluates causal discovery models by considering causal relationships, addressing limitations of traditional metrics like edit distance and KL divergence.

Contribution

The paper proposes Causal KL, a new metric that incorporates causal relationships to improve the evaluation of causal discovery models over existing metrics.

Findings

Causal KL effectively distinguishes between models with different causal claims.

Causal KL performs well in practical evaluations.

The augmented metric improves model assessment accuracy.

Abstract

The two most commonly used criteria for assessing causal model discovery with artificial data are edit-distance and Kullback-Leibler divergence, measured from the true model to the learned model. Both of these metrics maximally reward the true model. However, we argue that they are both insufficiently discriminating in judging the relative merits of false models. Edit distance, for example, fails to distinguish between strong and weak probabilistic dependencies. KL divergence, on the other hand, rewards equally all statistically equivalent models, regardless of their different causal claims. We propose an augmented KL divergence, which we call Causal KL (CKL), which takes into account causal relationships which distinguish between observationally equivalent models. Results are presented for three variants of CKL, showing that Causal KL works well in practice.