When is p-hacking detectable?

TL;DR

This paper introduces a novel statistical test to detect various forms of p-hacking by analyzing the distribution of reported t-statistics, addressing limitations of existing methods.

Contribution

The authors develop a sharp projection test that detects all forms of detectable p-hacking and demonstrate its effectiveness on real meta-analytic data.

Findings

The test can detect distortions in t-curves beyond chance or benign distortions.

Applied to a meta-dataset, the test finds significant distortions in RCTs and IV studies.

Standard tests may lack power to detect certain p-hacking practices.

Abstract

We show that some forms of p-hacking cannot be detected by examining the histogram of t-statistics or their p-values. Even when p-hacking is detectable, standard tests may lack power. We propose a novel test that detects every form of selective reporting that is detectable from the distribution of reported t-statistics. Our test statistic is the distance between the smoothed empirical t-curve and the set of possible honest distributions. This projection test is sharp and can only be evaded by selective reporting that also evades all other valid tests of restrictions on the t-curve. We also show how to avoid spurious rejections caused by some benign distortions in the t-curve. Applying the test to the Brodeur et al. (2020) meta-dataset, we find that the t-curves for RCTs and IVs are more distorted than could arise by chance, (de)rounding, or the Student-t approximation.