Can expected error costs justify testing a hypothesis at multiple alpha levels rather than searching for an elusive optimal alpha?

TL;DR

This paper explores how testing hypotheses at multiple alpha levels within a Neyman-Pearson framework can manage error costs effectively, offering a practical alternative to finding an optimal alpha, especially when model assumptions are uncertain.

Contribution

It introduces formulas for expected error costs in multi-alpha testing, compares them with single-alpha tests, and demonstrates the approach's robustness against model misspecification.

Findings

Multi-alpha testing can yield acceptable error costs.

The approach reduces the need for decision modeling.

Expected error costs may be lower than optimal models under uncertainty.

Abstract

Simultaneous testing of one hypothesis at multiple alpha levels can be performed within a conventional Neyman-Pearson framework. This is achieved by treating the hypothesis as a family of hypotheses, each member of which explicitly concerns test level as well as effect size. Such testing encourages researchers to think about error rates and strength of evidence in both the statistical design and reporting stages of a study. Here, we show that these multi-alpha level tests can deliver acceptable expected total error costs. We first present formulas for expected error costs from single alpha and multiple alpha level tests, given prior probabilities of effect sizes that have either dichotomous or continuous distributions. Error costs are tied to decisions, with different decisions assumed for each of the potential outcomes in the multi-alpha level case. Expected total costs for tests at single and multiple alpha levels are then compared with optimal costs. This comparison highlights how sensitive optimization is to estimated error costs and to assumptions about prevalence. Testing at multiple default thresholds removes the need to formally identify decisions, or to model costs and prevalence as required in optimization approaches. Although total expected error costs with this approach will not be optimal, our results suggest they may be lower, on average, than when so-called optimal test levels are based on mis-specified models.