The Power of Tests for Detecting $p$-Hacking

TL;DR

This paper analyzes the effectiveness of statistical tests in detecting p-hacking by examining how different hacking strategies influence p-value distributions and identifying the most powerful detection methods.

Contribution

It provides a theoretical assessment of the power of various tests for detecting p-hacking, highlighting the influence of hacking strategies and true effect distributions.

Findings

Power of detection tests varies with hacking strategies

Combined tests for bounds and monotonicity are most effective

Detection power depends on true effect distribution

Abstract

A flourishing empirical literature investigates the prevalence of $p$-hacking based on the distribution of $p$-values across studies. Interpreting results in this literature requires a careful understanding of the power of methods for detecting $p$-hacking. We theoretically study the implications of likely forms of $p$-hacking on the distribution of $p$-values to understand the power of tests for detecting it. Power can be low and depends crucially on the $p$-hacking strategy and the distribution of true effects. Combined tests for upper bounds and monotonicity and tests for continuity of the $p$-curve tend to have the highest power for detecting $p$-hacking.