Learning to Defer for Causal Discovery with Imperfect Experts

TL;DR

This paper introduces L2D-CD, a novel method that learns to intelligently defer between data-driven causal discovery and expert recommendations, improving accuracy when expert knowledge varies in reliability.

Contribution

The paper proposes a learning-to-defer approach tailored for causal discovery, effectively combining expert input with data-driven methods and identifying domains of expert strength or weakness.

Findings

L2D-CD outperforms standalone causal discovery and expert methods on Tübingen pairs dataset.

The approach accurately detects when expert advice is reliable or not.

It provides a foundation for extending causal discovery to larger variable graphs.

Abstract

Integrating expert knowledge, e.g. from large language models, into causal discovery algorithms can be challenging when the knowledge is not guaranteed to be correct. Expert recommendations may contradict data-driven results, and their reliability can vary significantly depending on the domain or specific query. Existing methods based on soft constraints or inconsistencies in predicted causal relationships fail to account for these variations in expertise. To remedy this, we propose L2D-CD, a method for gauging the correctness of expert recommendations and optimally combining them with data-driven causal discovery results. By adapting learning-to-defer (L2D) algorithms for pairwise causal discovery (CD), we learn a deferral function that selects whether to rely on classical causal discovery methods using numerical data or expert recommendations based on textual meta-data. We evaluate L2D-CD on the canonical T\"ubingen pairs dataset and demonstrate its superior performance compared to both the causal discovery method and the expert used in isolation. Moreover, our approach identifies domains where the expert's performance is strong or weak. Finally, we outline a strategy for generalizing this approach to causal discovery on graphs with more than two variables, paving the way for further research in this area.