Beyond Neyman-Pearson: e-values enable hypothesis testing with a data-driven alpha

TL;DR

This paper advocates for using e-values instead of p-values in hypothesis testing, highlighting their advantages in data-driven decision-making and risk control, especially in post-hoc and flexible settings.

Contribution

It introduces e-values as a superior alternative to p-values for hypothesis testing, enabling data-driven alpha levels and post-hoc risk management.

Findings

E-values provide straightforward Type-I risk control in generalized Neyman-Pearson settings.

E-values facilitate post-hoc decision rules with risk guarantees.

Powerful e-values exist for classical testing problems.

Abstract

A standard practice in statistical hypothesis testing is to mention the p-value alongside the accept/reject decision. We show the advantages of mentioning an e-value instead. With p-values, it is not clear how to use an extreme observation (e.g. p $\ll \alpha$) for getting better frequentist decisions. With e-values it is straightforward, since they provide Type-I risk control in a generalized Neyman-Pearson setting with the decision task (a general loss function) determined post-hoc, after observation of the data -- thereby providing a handle on `roving $\alpha$'s'. When Type-II risks are taken into consideration, the only admissible decision rules in the post-hoc setting turn out to be e-value-based. Similarly, if the loss incurred when specifying a faulty confidence interval is not fixed in advance, standard confidence intervals and distributions may fail whereas e-confidence sets and e-posteriors still provide valid risk guarantees. Sufficiently powerful e-values have by now been developed for a range of classical testing problems. We discuss the main challenges for wider development and deployment.