Publication Bias (The "File-Drawer Problem") in Scientific Inference

TL;DR

This paper examines publication bias, especially the file-drawer effect, highlighting its potential to significantly distort scientific results and criticizing the common Fail Safe File Drawer method for underestimating this bias.

Contribution

It demonstrates that small amounts of missing studies can cause large biases and shows that the widely used FSFD method often misestimates publication bias severity.

Findings

Small missing study numbers can cause significant bias

FSFD method often underestimates publication bias

Trust in statistical combination requires complete study inclusion

Abstract

Publication bias arises whenever the probability that a study is published depends on the statistical significance of its results. This bias, often called the file-drawer effect since the unpublished results are imagined to be tucked away in researchers' file cabinets, is potentially a severe impediment to combining the statistical results of studies collected from the literature. With almost any reasonable quantitative model for publication bias, only a small number of studies lost in the file-drawer will produce a significant bias. This result contradicts the well known Fail Safe File Drawer (FSFD) method for setting limits on the potential harm of publication bias, widely used in social, medical and psychic research. This method incorrectly treats the file drawer as unbiased, and almost always misestimates the seriousness of publication bias. A large body of not only psychic research, but medical and social science studies, has mistakenly relied on this method to validate claimed discoveries. Statistical combination can be trusted only if it is known with certainty that all studies that have been carried out are included. Such certainty is virtually impossible to achieve in literature surveys.