Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption

TL;DR

This paper advocates for rigorous synthetic experiments as essential for evaluating causal machine learning methods, aiming to enhance their reliability, trustworthiness, and adoption in real-world applications.

Contribution

It critically reviews current evaluation practices, highlights their shortcomings, and proposes principles for conducting rigorous synthetic experiments in causal machine learning.

Findings

Synthetic experiments are crucial for assessing causal methods.

Current evaluation practices are often inadequate.

Proposed principles improve reliability and trust in causal ML.

Abstract

Causal machine learning has the potential to revolutionize decision-making by combining the predictive power of machine learning algorithms with the theory of causal inference. However, these methods remain underutilized by the broader machine learning community, in part because current empirical evaluations do not permit assessment of their reliability and robustness, undermining their practical utility. Specifically, one of the principal criticisms made by the community is the extensive use of synthetic experiments. We argue, on the contrary, that synthetic experiments are essential and necessary to precisely assess and understand the capabilities of causal machine learning methods. To substantiate our position, we critically review the current evaluation practices, spotlight their shortcomings, and propose a set of principles for conducting rigorous empirical analyses with synthetic data. Adopting the proposed principles will enable comprehensive evaluations that build trust in causal machine learning methods, driving their broader adoption and impactful real-world use.