# A systematic approach to improving the reliability and scale of evidence from health care data

## TL;DR

This paper proposes a standardized, high-throughput approach to observational healthcare data analysis to improve evidence reliability, demonstrated by comparing depression treatments across multiple datasets with consistent methodology.

## Contribution

It introduces a paradigm shift towards high-throughput, standardized observational studies for more reliable and comprehensive healthcare evidence, moving away from isolated estimates.

## Key findings

- Good transitivity and consistency between databases
- Agreement with four out of five clinical trial findings
- More complete evidence with fewer small or null effects

## Abstract

Concerns over reproducibility in science extend to research using existing healthcare data; many observational studies investigating the same topic produce conflicting results, even when using the same data. To address this problem, we propose a paradigm shift. The current paradigm centers on generating one estimate at a time using a unique study design with unknown reliability and publishing (or not) one estimate at a time. The new paradigm advocates for high-throughput observational studies using consistent and standardized methods, allowing evaluation, calibration, and unbiased dissemination to generate a more reliable and complete evidence base. We demonstrate this new paradigm by comparing all depression treatments for a set of outcomes, producing 17,718 hazard ratios, each using methodology on par with state-of-the-art studies. We furthermore include control hypotheses to evaluate and calibrate our evidence generation process. Results show good transitivity and consistency between databases, and agree with four out of the five findings from clinical trials. The distribution of effect size estimates reported in literature reveals an absence of small or null effects, with a sharp cutoff at p = 0.05. No such phenomena were observed in our results, suggesting more complete and more reliable evidence.

---
Source: https://tomesphere.com/paper/1803.10791