Can expected error costs justify testing a hypothesis at multiple alpha levels rather than searching for an elusive optimal alpha?
Janet Aisbett, Stephan Leitner, Stephan Leitner, Stephan Leitner, Stephan Leitner
TL;DR
This paper explores whether testing a hypothesis at multiple alpha levels can reduce expected error costs compared to searching for a single optimal alpha.
Contribution
It introduces a method for multi-alpha level testing and compares its expected error costs to single-alpha and optimal approaches.
Findings
Multi-alpha level tests can yield acceptable expected total error costs.
Optimization of error costs is highly sensitive to assumptions about prevalence and cost estimates.
Using multiple default thresholds may lead to lower average error costs than mis-specified optimal models.
Abstract
Simultaneous testing of one hypothesis at multiple alpha levels can be performed within a conventional Neyman-Pearson framework. This is achieved by treating the hypothesis as a family of hypotheses, each member of which explicitly concerns test level as well as effect size. Such testing encourages researchers to think about error rates and strength of evidence in both the statistical design and reporting stages of a study. Here, we show that these multi-alpha level tests can deliver acceptable expected total error costs. We first present formulas for expected error costs from single alpha and multiple alpha level tests, given prior probabilities of effect sizes that have either dichotomous or continuous distributions. Error costs are tied to decisions, with different decisions assumed for each of the potential outcomes in the multi-alpha level case. Expected total costs for tests at…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStatistical Methods in Clinical Trials · Statistical Methods and Bayesian Inference · Advanced Causal Inference Techniques