Significance testing without truth

TL;DR

This paper argues that significance testing should focus on data consistency with hypotheses rather than on their truthfulness, emphasizing practical utility over strict truth evaluation.

Contribution

It clarifies the distinction between testing for data inconsistency and asserting the truth of models, proposing a pragmatic approach to significance testing.

Findings

Significance tests assess data-model consistency, not model truth.

Many models are inherently false but still useful for testing.

Tests reject hypotheses only when data is highly improbable under the model.

Abstract

A popular approach to significance testing proposes to decide whether the given hypothesized statistical model is likely to be true (or false). Statistical decision theory provides a basis for this approach by requiring every significance test to make a decision about the truth of the hypothesis/model under consideration. Unfortunately, many interesting and useful models are obviously false (that is, not exactly true) even before considering any data. Fortunately, in practice a significance test need only gauge the consistency (or inconsistency) of the observed data with the assumed hypothesis/model -- without enquiring as to whether the assumption is likely to be true (or false), or whether some alternative is likely to be true (or false). In this practical formulation, a significance test rejects a hypothesis/model only if the observed data is highly improbable when calculating the probability while assuming the hypothesis being tested; the significance test only gauges whether the observed data likely invalidates the assumed hypothesis, and cannot decide that the assumption -- however unmistakably false -- is likely to be false a priori, without any data.