A proposal for informative default priors scaled by the standard error of estimates

TL;DR

This paper proposes a method for setting informative default priors scaled by the standard error, using a large corpus of studies to improve effect size estimates and address biases like the winner's curse.

Contribution

It introduces a novel approach to estimate priors from study corpora, enhancing regularization and reducing bias in effect size estimation.

Findings

Demonstrated with psychology and clinical trial data

Reduces effect size overestimation compared to uniform priors

Improves long-term inference stability

Abstract

If we have an unbiased estimate of some parameter of interest, then its absolute value is positively biased for the absolute value of the parameter. This bias is large when the signal-to-noise ratio (SNR) is small, and it becomes even larger when we condition on statistical significance; the winner's curse. This is a frequentist motivation for regularization. To determine a suitable amount of shrinkage, we propose to estimate the distribution of the SNR from a large collection or corpus of similar studies and use this as a prior distribution. The wider the scope of the corpus, the less informative the prior, but a wider scope does not necessarily result in a more diffuse prior. We show that the estimation of the prior simplifies if we require that posterior inference is equivariant under linear transformations of the data. We demonstrate our approach with corpora of 86 replication studies from psychology and 178 phase 3 clinical trials. Our suggestion is not intended to be a replacement for a prior based on full information about a particular problem; rather, it represents a familywise choice that should yield better long-term properties than the current default uniform prior, which has led to systematic overestimates of effect sizes and a replication crisis when these inflated estimates have not shown up in later studies.