A systematic approach to improving the reliability and scale of evidence from health care data

TL;DR

This paper proposes a standardized, high-throughput approach to observational healthcare data analysis to improve evidence reliability, demonstrated by comparing depression treatments across multiple datasets with consistent methodology.

Contribution

It introduces a paradigm shift towards high-throughput, standardized observational studies for more reliable and comprehensive healthcare evidence, moving away from isolated estimates.

Findings

Good transitivity and consistency between databases

Agreement with four out of five clinical trial findings

More complete evidence with fewer small or null effects

Abstract

Concerns over reproducibility in science extend to research using existing healthcare data; many observational studies investigating the same topic produce conflicting results, even when using the same data. To address this problem, we propose a paradigm shift. The current paradigm centers on generating one estimate at a time using a unique study design with unknown reliability and publishing (or not) one estimate at a time. The new paradigm advocates for high-throughput observational studies using consistent and standardized methods, allowing evaluation, calibration, and unbiased dissemination to generate a more reliable and complete evidence base. We demonstrate this new paradigm by comparing all depression treatments for a set of outcomes, producing 17,718 hazard ratios, each using methodology on par with state-of-the-art studies. We furthermore include control hypotheses to evaluate and calibrate our evidence generation process. Results show good transitivity and consistency between databases, and agree with four out of the five findings from clinical trials. The distribution of effect size estimates reported in literature reveals an absence of small or null effects, with a sharp cutoff at p = 0.05. No such phenomena were observed in our results, suggesting more complete and more reliable evidence.