OCDB: Revisiting Causal Discovery with a Comprehensive Benchmark and Evaluation Framework

TL;DR

This paper introduces OCDB, a comprehensive benchmark and evaluation framework for causal discovery that emphasizes interpretability and real-world data assessment, revealing gaps in current algorithms.

Contribution

We propose a flexible evaluation framework with new metrics and introduce OCDB, a real-data benchmark to improve fairness and effectiveness in causal discovery evaluations.

Findings

Existing algorithms show limited generalization on real data.

The new metrics enable fair comparisons between different causal graph types.

Our framework highlights the need for improved causal discovery methods.

Abstract

Large language models (LLMs) have excelled in various natural language processing tasks, but challenges in interpretability and trustworthiness persist, limiting their use in high-stakes fields. Causal discovery offers a promising approach to improve transparency and reliability. However, current evaluations are often one-sided and lack assessments focused on interpretability performance. Additionally, these evaluations rely on synthetic data and lack comprehensive assessments of real-world datasets. These lead to promising methods potentially being overlooked. To address these issues, we propose a flexible evaluation framework with metrics for evaluating differences in causal structures and causal effects, which are crucial attributes that help improve the interpretability of LLMs. We introduce the Open Causal Discovery Benchmark (OCDB), based on real data, to promote fair comparisons and drive optimization of algorithms. Additionally, our new metrics account for undirected edges, enabling fair comparisons between Directed Acyclic Graphs (DAGs) and Completed Partially Directed Acyclic Graphs (CPDAGs). Experimental results show significant shortcomings in existing algorithms' generalization capabilities on real data, highlighting the potential for performance improvement and the importance of our framework in advancing causal discovery techniques.