Predicting treatment effects from observational studies using machine learning methods: A simulation study

TL;DR

This study evaluates machine learning methods for estimating treatment effects from observational data through simulations, revealing they perform well with linear relationships but poorly with non-linear data, especially under confounding.

Contribution

It provides a systematic simulation-based assessment of machine learning counterfactual prediction methods, highlighting their limitations with non-linear data and confounding.

Findings

Machine learning models perform well on linear data.

Performance drops significantly with non-linear relationships.

Confounding does not significantly affect linear data performance.

Abstract

Measuring treatment effects in observational studies is challenging because of confounding bias. Confounding occurs when a variable affects both the treatment and the outcome. Traditional methods such as propensity score matching estimate treatment effects by conditioning on the confounders. Recent literature has presented new methods that use machine learning to predict the counterfactuals in observational studies which then allow for estimating treatment effects. These studies however, have been applied to real world data where the true treatment effects have not been known. This study aimed to study the effectiveness of this counterfactual prediction method by simulating two main scenarios: with and without confounding. Each type also included linear and non-linear relationships between input and output data. The key item in the simulations was that we generated known true causal effects. Linear regression, lasso regression and random forest models were used to predict the counterfactuals and treatment effects. These were compared these with the true treatment effect as well as a naive treatment effect. The results show that the most important factor in whether this machine learning method performs well, is the degree of non-linearity in the data. Surprisingly, for both non-confounding \textit{and} confounding, the machine learning models all performed well on the linear dataset. However, when non-linearity was introduced, the models performed very poorly. Therefore under the conditions of this simulation study, the machine learning method performs well under conditions of linearity, even if confounding is present, but at this stage should not be trusted when non-linearity is introduced.