When More Is Less: Pitfalls of significance testing

TL;DR

This paper discusses the longstanding controversy over the use of significance testing, highlighting potential pitfalls and limitations that can impact scientific conclusions across various fields.

Contribution

It provides a critical analysis of significance testing, illustrating common pitfalls and encouraging more nuanced interpretation of statistical results.

Findings

Significance testing can lead to misleading conclusions.

Small p-values do not necessarily imply practical significance.

Misinterpretation of significance tests is widespread across disciplines.

Abstract

The controversy about statistical significance vs. scientific relevance is more than 100 years old. But still nowadays null hypothesis significance testing is considered as gold standard in many empirical fields from economics and social sciences over psychology to medicine, and small $p$-values are often the key to publish in journals of high scientific reputation. I highlight, illustrate and discuss potential pitfalls of statistical significance testing on three occasions.