Defending the P-value

TL;DR

This paper defends the P-value against recent criticisms, arguing it remains a crucial, context-dependent tool for balancing false positives and negatives in scientific research, rather than a flawed or obsolete measure.

Contribution

The paper provides a detailed argument supporting the continued use of the P-value, emphasizing its importance as a calibrated, tunable tool for error balancing in scientific inference.

Findings

P-values help control error trade-offs in research.

Banning P-values would remove a valuable, adjustable tool.

Context-specific interpretation of P-values is essential.

Abstract

Attacks on the P-value are nothing new, but the recent attacks are increasingly more serious. They come from more mainstream sources, with widening targets such as a call to retire the significance testing altogether. While well meaning, I believe these attacks are nevertheless misdirected: Blaming the P-value for the naturally tentative trial-and-error process of scientific discoveries, and presuming that banning the P-value would make the process cleaner and less error-prone. However tentative, the skeptical scientists still have to form unambiguous opinions, proximately to move forward in their investigations and ultimately to present results to the wider community. With obvious reasons, they constantly need to balance between the false-positive and false-negative errors. How would banning the P-value or significance tests help in this balancing act? It seems trite to say that this balance will always depend on the relative costs or the trade-off between the errors. These costs are highly context specific, varying by area of applications or by stage of investigation. A calibrated but tunable knob, such as that given by the P-value, is needed for controlling this balance. This paper presents detailed arguments in support of the P-value.