On admissibility in post-hoc hypothesis testing

TL;DR

This paper develops a theory for post-hoc hypothesis testing that allows significance levels to be chosen after data analysis, introducing $ ext{Gamma}$-admissibility and classifying optimal rules based on e-values.

Contribution

It introduces a formal framework for post-hoc hypothesis testing, extending classical methods to allow adaptive significance levels based on data.

Findings

$ ext{Gamma}$-admissible rules are based on e-values.

Classifies $ ext{Gamma}$-admissible rules for various adversary sets.

Recovers Neyman-Pearson lemma for constant significance levels.

Abstract

The validity of classical hypothesis testing requires the significance level $\alpha$ be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating $\alpha$ during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value $p\ll\alpha$ vs $p= \alpha- \epsilon$ for tiny $\epsilon > 0$ has no (statistical) relevance for any downstream decision-making. Following recent work of Gr\"unwald (2024), we develop a theory of post-hoc hypothesis testing, enabling $\alpha$ to be chosen after seeing and analyzing the data. To study "good" post-hoc tests we introduce $\Gamma$-admissibility, where $\Gamma$ is a set of adversaries which map the data to a significance level. We classify the set of $\Gamma$-admissible rules for various sets $\Gamma$, showing they must be based on e-values, and recover the Neyman-Pearson lemma when $\Gamma$ is the constant map.