Assessing Treatment Effect Variation in Observational Studies: Results from a Data Challenge

TL;DR

This paper reports on a workshop where researchers compared various methods for estimating how treatment effects vary in observational studies, highlighting the challenges and differences in results.

Contribution

It provides a comparative analysis of multiple methods for assessing treatment effect heterogeneity using synthetic data from a large-scale educational trial.

Findings

Broad consensus on average treatment effect estimate

Large differences in estimated treatment effect moderation

Highlights challenges in estimating treatment effect variation

Abstract

A growing number of methods aim to assess the challenging question of treatment effect variation in observational studies. This special section of "Observational Studies" reports the results of a workshop conducted at the 2018 Atlantic Causal Inference Conference designed to understand the similarities and differences across these methods. We invited eight groups of researchers to analyze a synthetic observational data set that was generated using a recent large-scale randomized trial in education. Overall, participants employed a diverse set of methods, ranging from matching and flexible outcome modeling to semiparametric estimation and ensemble approaches. While there was broad consensus on the topline estimate, there were also large differences in estimated treatment effect moderation. This highlights the fact that estimating varying treatment effects in observational studies is often more challenging than estimating the average treatment effect alone. We suggest several directions for future work arising from this workshop.

Figures