How balance and sample size impact bias in the estimation of causal treatment effects: A simulation study

TL;DR

This simulation study investigates how balance, sample size, and other factors affect bias in estimating causal treatment effects using observational data, providing practical guidelines for reliable inference.

Contribution

It offers new insights into the importance of balance assessment and sample size requirements for unbiased causal effect estimation using propensity score methods.

Findings

Maximum Kolmogorov-Smirnov statistic is effective for balance assessment.

An acceptable balance threshold is a KS statistic of 0.1.

A sample size of 60-80 observations per confounder per group is recommended.

Abstract

Observational studies are often used to understand relationships between exposures and outcomes. They do not, however, allow conclusions about causal relationships to be drawn unless statistical techniques are used to account for the imbalance of confounders across exposure groups. Propensity score and balance weighting (PSBW) are useful techniques that aim to reduce the imbalances between exposure groups by weighting the groups to look alike on the observed confounders. Despite the plethora of available methods to estimate PSBW, there is little guidance on what one defines as adequate balance, and unbiased and robust estimation of the causal treatment effect is not guaranteed unless several conditions hold. Accurate inference requires that 1. the treatment allocation mechanism is known, 2. the relationship between the baseline covariates and the outcome is known, 3. adequate balance of baseline covariates is achieved post-weighting, 4. a proper set of covariates to control for confounding bias is known, and 5. a large enough sample size is available. In this article, we use simulated data of various sizes to investigate the influence of these five factors on statistical inference. Our findings provide evidence that the maximum Kolmogorov- Smirnov statistic is the proper statistical measure to assess balance on the baseline covariates, in contrast to the mean standardised mean difference used in many applications, and 0.1 is a suitable threshold to consider as acceptable balance. Finally, we recommend that 60-80 observations, per confounder per treatment group, are required to obtain a reliable and unbiased estimation of the causal treatment effect.