How do dataset characteristics affect the performance of propensity score methods and regression for controlling confounding in observational studies? A simulation study

TL;DR

This simulation study compares propensity score methods and regression in observational studies, revealing how dataset features like size, overlap, and prevalence impact their effectiveness in controlling confounding.

Contribution

It provides a systematic evaluation of how dataset characteristics influence the performance of propensity score methods versus regression, guiding method selection.

Findings

Regression performs better with large samples and low imbalance.

Propensity score methods struggle with poor overlap and small samples.

Inverse probability weighting shows bias with low overlap.

Abstract

In observational studies, researchers must select a method to control for confounding. Options include propensity score methods and regression. It remains unclear how dataset characteristics (size, overlap in propensity scores, exposure prevalence) influence the relative performance of the methods, making it difficult to select the best method for a particular dataset. A simulation study to evaluate the role of dataset characteristics on the performance of propensity score methods, compared to logistic regression, for estimating a marginal odds ratio in the presence of confounding was conducted. Outcomes were simulated from logistic and complementary log-log models, and size, overlap in propensity scores, and prevalence of the exposure were varied. Regression showed poor coverage for small sample sizes, but with large sample sizes it was more robust to imbalance in propensity scores and low exposure prevalence than were propensity score methods. Propensity score methods frequently displayed suboptimal coverage, particularly as overlap in propensity scores decreased. These problems were exacerbated at larger sample sizes. Power of matching methods was particularly affected by lack of overlap, low prevalence of exposure, and small sample size. Performance of inverse probability of treatment weighting depended heavily on dataset characteristics, with poor coverage and bias with low overlap. The advantage of regression for large data size was less clear in sensitivity analysis with a complementary log-log outcome generation mechanism and unmeasured confounding, with superior bias and error but lower coverage than nearest neighbour and caliper matching.