The statistical properties of RCTs and a proposal for shrinkage

TL;DR

This paper analyzes the statistical properties of randomized controlled trials (RCTs), estimates the distribution of effect sizes and power from a large dataset, and proposes a shrinkage method to correct for overestimation bias.

Contribution

It introduces a statistical framework for RCTs, estimates key distributions from extensive data, and proposes a novel shrinkage method to reduce effect size exaggeration.

Findings

Median achieved power is 0.13

Effect size overestimation factor is 1.7 for just significant results

Proposes a shrinkage method to improve effect size estimates

Abstract

We abstract the concept of a randomized controlled trial (RCT) as a triple (beta,b,s), where beta is the primary efficacy parameter, b the estimate and s the standard error (s>0). The parameter beta is either a difference of means, a log odds ratio or a log hazard ratio. If we assume that b is unbiased and normally distributed, then we can estimate the full joint distribution of (beta,b,s) from a sample of pairs (b_i,s_i). We have collected 23,747 such pairs from the Cochrane database to do so. Here, we report the estimated distribution of the signal-to-noise ratio beta/s and the achieved power. We estimate the median achieved power to be 0.13. We also consider the exaggeration ratio which is the factor by which the magnitude of beta is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so.