Comparing Propensity Score-Based Methods in Estimating the Treatment Effects: A Simulation Study

TL;DR

This study compares various propensity score methods in estimating treatment effects through simulations, highlighting their strengths and robustness across different scenarios.

Contribution

It introduces a comprehensive simulation framework to evaluate and compare propensity score methods for treatment effect estimation.

Findings

Inverse probability weighting yields estimates closer to the true effect.

Matching and stratification better control the validity domain of estimates.

Performance varies with scenario complexity and method used.

Abstract

In observational studies, the recorded treatment assignment is not purely random, but it is influenced by external factors such as patient characteristics, reimbursement policies, and existing guidelines. Therefore, the treatment effect can be estimated only after accounting for confounding factors. Propensity score (PS) methods are a family of methods that is widely used for this purpose. Although they are all based on the estimation of the a posteriori probability of treatment assignment given patient covariates, they estimate the treatment effect from different statistical points of view and are, thus, relatively hard to compare. In this work, we propose a simulation experiment in which a hypothetical cohort of subjects is simulated in seven scenarios of increasing complexity of the associations between covariates and treatment, but where the two main definitions of treatment effect (average treatment effect, ATE, and average effect of the treatment on the treated, ATT) coincide. Our purpose is to compare the performance of a wide array of PS-based methods (matching, stratification, and inverse probability weighting) in estimating the treatment effect and their robustness in different scenarios. We find that inverse probability weighting provides estimates of the treatment effect that are closer to the expected value by weighting all subjects of the starting population. Conversely, matching and stratification ensure that the subpopulation that generated the final estimate is made up of real instances drawn from the starting population, and, thus, provide a higher degree of control on the validity domain of the estimates.