Causal Machine Learning Methods for Estimating Personalised Treatment Effects -- Insights on validity from two large trials

TL;DR

This study critically evaluates 17 causal ML methods for personalized treatment effect estimation using large clinical trial data, revealing significant validation issues and limited generalizability, thus questioning their readiness for precision medicine.

Contribution

It provides the first comprehensive empirical validation of causal ML methods on large RCT datasets, highlighting their reliability issues in real-world settings.

Findings

None of the methods reliably validated performance internally or externally.

Significant discrepancies between training and test data were observed.

Estimated treatment effects did not generalize across datasets.

Abstract

Causal machine learning (ML) methods hold great promise for advancing precision medicine by estimating personalized treatment effects. However, their reliability remains largely unvalidated in empirical settings. In this study, we assessed the internal and external validity of 17 mainstream causal heterogeneity ML methods -- including metalearners, tree-based methods, and deep learning methods -- using data from two large randomized controlled trials: the International Stroke Trial (N=19,435) and the Chinese Acute Stroke Trial (N=21,106). Our findings reveal that none of the ML methods reliably validated their performance, neither internal nor external, showing significant discrepancies between training and test data on the proposed evaluation metrics. The individualized treatment effects estimated from training data failed to generalize to the test data, even in the absence of distribution shifts. These results raise concerns about the current applicability of causal ML models in precision medicine, and highlight the need for more robust validation techniques to ensure generalizability.