P-values: misunderstood and misused

TL;DR

This paper reviews critical perspectives on p-values, clarifies misconceptions, and offers practical recommendations for their proper use and interpretation in scientific research.

Contribution

It synthesizes recent critiques of p-values, explains differences with false discovery rates, and proposes practical steps to improve their application in research.

Findings

P-values are often misunderstood and misused in scientific studies.

Lower significance thresholds like 0.01 or 0.001 are recommended.

Contextual interpretation of p-values enhances research validity.

Abstract

P-values are widely used in both the social and natural sciences to quantify the statistical significance of observed results. The recent surge of big data research has made the p-value an even more popular tool to test the significance of a study. However, substantial literature has been produced critiquing how p-values are used and understood. In this paper we review this recent critical literature, much of which is routed in the life sciences, and consider its implications for social scientific research. We provide a coherent picture of what the main criticisms are, and draw together and disambiguate common themes. In particular, we explain how the False Discovery Rate is calculated, and how this differs from a p-value. We also make explicit the Bayesian nature of many recent criticisms, a dimension that is often underplayed or ignored. We conclude by identifying practical steps to help remediate some of the concerns identified. We recommend that (i) far lower significance levels are used, such as $0.01$ or $0.001$, and (ii) p-values are interpreted contextually, and situated within both the findings of the individual study and the broader field of inquiry (through, for example, meta-analyses).