Causal Discovery with Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning-based method for causal discovery that improves search efficiency and flexibility in identifying directed acyclic graphs from data, outperforming traditional heuristics.

Contribution

The paper proposes a novel RL-based approach for causal discovery that uses neural models to generate graph structures, enhancing search capability and accommodating flexible scoring functions.

Findings

Improved accuracy in causal graph identification on synthetic datasets.

Effective application to real-world datasets demonstrating practical utility.

Enhanced search efficiency over traditional score-based methods.

Abstract

Discovering causal structure among a set of variables is a fundamental problem in many empirical sciences. Traditional score-based casual discovery methods rely on various local heuristics to search for a Directed Acyclic Graph (DAG) according to a predefined score function. While these methods, e.g., greedy equivalence search, may have attractive results with infinite samples and certain model assumptions, they are usually less satisfactory in practice due to finite data and possible violation of assumptions. Motivated by recent advances in neural combinatorial optimization, we propose to use Reinforcement Learning (RL) to search for the DAG with the best scoring. Our encoder-decoder model takes observable data as input and generates graph adjacency matrices that are used to compute rewards. The reward incorporates both the predefined score function and two penalty terms for enforcing acyclicity. In contrast with typical RL applications where the goal is to learn a policy, we use RL as a search strategy and our final output would be the graph, among all graphs generated during training, that achieves the best reward. We conduct experiments on both synthetic and real datasets, and show that the proposed approach not only has an improved search ability but also allows a flexible score function under the acyclicity constraint.