Confidence intervals with maximal average power

TL;DR

This paper introduces a frequentist testing method that maximizes power while maintaining coverage by incorporating a prior distribution, offering a tunable and efficient approach for hypothesis testing.

Contribution

It presents a novel frequentist test that leverages Bayesian posterior distributions to optimize power based on a prior, balancing flexibility and simplicity.

Findings

Enhanced power when data aligns with the prior

Simple implementation without minimax optimization

Illustration with binomial experiments

Abstract

We propose a frequentist testing procedure that maintains a defined coverage and is optimal in the sense that it gives maximal power to detect deviations from a null hypothesis when the alternative to the null hypothesis is sampled from a pre-specified distribution (the prior distribution). Selecting a prior distribution allows to tune the decision rule. This leads to an increased power, if the true data generating distribution happens to be compatible with the prior. It comes at the cost of losing power, if the data generating distribution or the observed data are incompatible with the prior. We illustrate the proposed approach for a binomial experiment, which is sufficiently simple such that the decision sets can be illustrated in figures, which should facilitate an intuitive understanding. The potential beyond the simple example will be discussed: the approach is generic in that the test is defined based on the likelihood function and the prior only. It is comparatively simple to implement and efficient to execute, since it does not rely on Minimax optimization. Conceptually it is interesting to note that for constructing the testing procedure the Bayesian posterior probability distribution is used.