Nested sampling for frequentist computation: fast estimation of small $p$-values

TL;DR

This paper introduces a nested sampling method for efficiently estimating small p-values in frequentist statistics, significantly reducing computational costs compared to traditional Monte Carlo methods, especially for high significance levels.

Contribution

It adapts nested sampling for frequentist p-value computation, demonstrating faster estimation for small p-values and establishing new links between Bayesian and frequentist approaches.

Findings

NS scales as log^2(1/p), outperforming 1/p of MC

Requires fewer simulations for >4σ significances

Potential for improved tuning of NS implementations

Abstract

We propose a novel method for computing $p$-values based on nested sampling (NS) applied to the sampling space rather than the parameter space of the problem, in contrast to its usage in Bayesian computation. The computational cost of NS scales as $\log^2{1/p}$, which compares favorably to the $1/p$ scaling for Monte Carlo (MC) simulations. For significances greater than about $4\sigma$ in both a toy problem and a simplified resonance search, we show that NS requires orders of magnitude fewer simulations than ordinary MC estimates. This is particularly relevant for high-energy physics, which adopts a $5\sigma$ gold standard for discovery. We conclude with remarks on new connections between Bayesian and frequentist computation and possibilities for tuning NS implementations for still better performance in this setting.