Thou Shalt Not Reject the P-value

TL;DR

This paper critically examines the historical and current use of P-values in scientific research, discussing their benefits, misuses, and alternatives, and offers recommendations for more reliable interpretation and application.

Contribution

It provides a comprehensive review of P-value usage, compares it with Bayesian methods, and discusses meta-analysis, emphasizing context-aware interpretation and proper application.

Findings

P-values are useful but often misinterpreted.

Bayesian alternatives offer different evidence-seeking approaches.

Meta-analysis can aggregate evidence but has limitations.

Abstract

Since its debut in the 18th century, the P-value has been an important part of hypothesis testing-based scientific discoveries. As the statistical engine accelerates, questions are beginning to be raised, asking to what extent scientific discoveries based on P-values are reliable and reproducible, and the voice calling for adjusting the significance level or banning the P-value has been increasingly heard. Inspired by these questions and discussions, here we enquire into the useful roles and misuses of the P-value in scientific studies. For common misuses and misinterpretations, we provide modest recommendations for practitioners. Additionally, we compare statistical significance with clinical relevance. In parallel, we review the Bayesian alternatives for seeking evidence. Finally, we discuss the promises and risks of using meta-analysis to pool P-values from multiple studies to aggregate evidence. Taken together, the P-value underpins a useful probabilistic decision-making system and provides evidence at a continuous scale. But its interpretation must be contextual, considering the scientific question, experimental design (including the model specification, sample size, and significance level), statistical power, effect size, and reproducibility.