# Estimating real-world treatment effects in the presence of measurement error and sparse outcome data using propensity score methods

## TL;DR

This study compares how well different statistical methods can estimate treatment effects from real-world health data, even when data is incomplete or has errors.

## Contribution

The study evaluates the performance of four propensity score methods under realistic conditions of measurement error and sparse data in EHRs.

## Key findings

- 3:1 PS matching had the largest bias in treatment effect estimates compared to other PS methods.
- Covariate under-recording increased mean squared error, while over-recording reduced it significantly.
- Outcome prevalence differences showed varying impacts on bias across PS methods.

## Abstract

The real-world treatment effect of a novel treatment can be estimated by analysing routinely collected patient data, in the form of Electronic Health Records (EHR). Any treatment allocation in EHR is not randomised and there may be systematic differences between the treatment groups. Propensity Score (PS) methods are commonly used to correct for these differences and reduce the bias in the treatment effect estimate. The aims of the study were to compare the performance of the most popular PS methods in the estimation of the treatment effect in the presence of two common issues in EHRs: covariate measurement error and sparse data.

The motivational example for this study was the assessment of the treatment effect of the novel oral anti-coagulant Rivaroxaban compared with the previous standard treatment Warfarin for the prevention of future stroke in patients with atrial fibrillation. Using simulation experiments based on a dataset comparing Rivaroxaban with Warfarin, we evaluated the performance of four PS methods.

In the simulations with characteristics of the original dataset, using 3:1 PS matching generated a largest bias of +0.0428 (corresponding ratio of HRs (rHR) 1.0437), whereas for the other PS methods it was smaller and in negative direction: IPTW for ATE -0.0181 (rHR = 0.9821); IPTW for ATT -0.0110 (rHR = 0.9891); PS stratification −0.0099 (rHR = 0.9901), with relative differences between rHRs being small to negligible. Fifty percent under-recording of a covariate (stroke) in the PS model, increased the MSE between 6% and 11% compared to the MSE with no introduced measurement error. While 50% over-recording reduced the MSE by around 35%. The difference in the bias of the low prevalence outcome (0.5%) and the high prevalence outcome (10%) was: IPTW for ATE 0.1514 (rHRs = 1.1635); IPTW for ATT 0.0160 (rHRs = 1.0161); 3:1 PS matching 0.0758 (rHRs = 1.0787); PS Stratification 0.0177 (rHRs = 1.0179). A similar pattern for outcome prevalence was seen for all the simulation scenarios.

This study showed that PS methods proposed in the literature may not all perform well for individual datasets. The findings produced recommendations for using PS methods in the estimation of real-world treatment effect when the covariate measurement error and sparse outcome data are present.