Post-Hoc Large-Sample Statistical Inference

TL;DR

This paper develops asymptotic post-hoc inference methods that allow for data-dependent significance levels, providing more flexible and sharper confidence sets and p-values than existing nonasymptotic approaches.

Contribution

It introduces a theory of asymptotic post-hoc inference, relaxing assumptions and improving the sharpness of confidence sets and p-values compared to prior nonasymptotic methods.

Findings

Asymptotic post-hoc confidence sets are valid under weaker assumptions.

Asymptotic p-values are sharper and more flexible.

The methods outperform nonasymptotic counterparts in large-sample scenarios.

Abstract

We derive inferential procedures for large sample sizes that remain valid under data-dependent significance levels (so-called "post-hoc valid inference"). Classical statistical tools require that the significance level -- the "type-I error" -- is selected prior to seeing or analyzing any data. This restriction leads to some drawbacks. For instance, if an analyst generates an inconclusive confidence interval, repeating the process with a larger significance level is not an option -- the result is final. Recently, e-values have emerged as the solution to this problem, being both necessary and sufficient tools for performing various forms of post-hoc inference. All such results, however, have thus far been nonasymptotic. As a result, they inherit familiar limitations of nonasymptotic inferential procedures such as requiring strong moment assumptions and being conservative in general. This paper develops a theory of post-hoc inference in the asymptotic setting, yielding asymptotic post-hoc confidence sets and asymptotic post-hoc p-values that make weaker assumptions and are sharper than their nonasymptotic counterparts.