A five-decision testing procedure to infer on unidimensional parameter

TL;DR

This paper introduces a five-decision testing procedure that combines the strengths of existing methods, enabling more efficient hypothesis testing with reduced sample sizes while accommodating plausible point hypotheses.

Contribution

It proposes a novel five-decision rule testing procedure that improves power and reduces sample size requirements compared to traditional and existing multi-decision tests.

Findings

Reduces sample size needed for statistical significance by about 20%.

Combines advantages of Kaiser and Jones-Tukey procedures.

Applicable when point null hypotheses are plausible.

Abstract

A statistical test can be seen as a procedure to produce a decision based on observed data, where some decisions consist of rejecting a hypothesis (yielding a significant result) and some do not, and where one controls the probability to make a wrong rejection at some pre-specified significance level. Whereas traditional hypothesis testing involves only two possible decisions (to reject or not a null hypothesis), Kaiser's directional two-sided test as well as the more recently introduced Jones and Tukey's testing procedure involve three possible decisions to infer on unidimensional parameter. The latter procedure assumes that a point null hypothesis is impossible (e.g. that two treatments cannot have exactly the same effect), allowing a gain of statistical power. There are however situations where a point hypothesis is indeed plausible, for example when considering hypotheses derived from Einstein's theories. In this article, we introduce a five-decision rule testing procedure, which combines the advantages of the testing procedures of Kaiser (no assumption on a point hypothesis being impossible) and of Jones and Tukey (higher power), allowing for a non-negligible (typically 20%) reduction of the sample size needed to reach a given statistical power to get a significant result, compared to the traditional approach.