Distributional Null Hypothesis Testing with the T distribution

TL;DR

This paper advocates for using distributional null hypotheses in NHST, which improves the validity and replicability of statistical testing compared to traditional point-form nulls, addressing many existing problems.

Contribution

It introduces and justifies the use of distributional nulls in NHST, providing a better-motivated and more reliable framework for hypothesis testing.

Findings

Distributional nulls are mathematically and experimentally better justified.

Using distributional nulls improves the validity of NHST.

Distributional nulls account for replication probability in significance testing.

Abstract

Null Hypothesis Significance Testing (NHST) has long been central to the scientific project, guiding theory development and supporting evidence-based intervention and decision-making. Recent years, however, have seen growing awareness of serious problems with NHST as it is typically used, and hence to proposals to limit the use of NHST techniques, to abandon these techniques and move to alternative statistical approaches, or even to ban the use of NHST entirely. These proposals are premature, because the observed problems with NHST all arise as a consequence of a contingent and in many cases incorrect choice: that of NHST testing against point-form nulls. We show that testing against distributional, rather than point-form, nulls is better motivated mathematically and experimentally, and that the use of distributional nulls addresses many problems with the standard point-form NHST approach. We also show that use of distributional nulls allows a form of null hypothesis testing that takes into account both the statistical significance of a given result and the probability of replication of that result in a new experiment. Rather than abandoning NHST, we should use the NHST approach in its more general form, with distributional rather than point-form nulls.