Significance and Replication in simple counting experiments: Distributional Null Hypothesis Testing

TL;DR

This paper advocates for replacing point-form null hypotheses in NHST with distributional nulls, which improves the validity and replicability of statistical testing in psychology and other sciences.

Contribution

It introduces and advocates for a distributional null hypothesis testing approach as a more valid alternative to traditional point-form NHST, with practical and theoretical advantages.

Findings

Distributional nulls address sample size issues.

The approach allows better estimation of replication probability.

It provides a mathematically sound alternative to standard NHST.

Abstract

Null Hypothesis Significance Testing (NHST) has long been of central importance to psychology as a science, guiding theory development and underlying the application of evidence-based intervention and decision-making. Recent years, however, have seen growing awareness of serious problems with NHST as it is typically used; this awareness has led to proposals to limit the use of NHST techniques, to abandon these techniques and move to alternative statistical approaches, or even to ban the use of NHST entirely. These proposals are premature, because the observed problems with NHST arise as a consequence of an historically contingent, essentially unmotivated, and fundamentally incorrect, choice: that of NHST testing against point-form null hypotheses. Using simple counting experiments we give a detailed presentation of an alternative, more general approach: that of testing against distributional nulls. We show that this distributional approach is well-motivated mathematically, practically and experimentally, and that the use of distributional nulls addresses various problems with the standard point-form NHST approach, avoiding issues to do with sample size and allowing a coherent estimation of the probability of replication of a given experimental result. Rather than abandoning NHST, we should use the NHST approach in its most general form, with distributional rather than point-form null hypotheses.