Why Empirical p-Values Are Not Uniform: Reference Samples, Dependence, and PIT Backtesting

TL;DR

This paper reveals that empirical p-values and PITs, when estimated from finite samples, deviate from uniformity due to dependence and variance distortions, impacting calibration assessment methods.

Contribution

It demonstrates how common implementations alter the statistical properties of empirical p-values, necessitating revised calibration techniques.

Findings

Empirical p-values are not uniformly distributed under finite samples.

Dependence and variance distortions invalidate classical uniformity tests.

Backtesting procedures need adjustments to account for two-stage sampling.

Abstract

Probability integral transforms (PITs) and empirical $p$-values are widely used to assess the calibration of predictive distributions. While exact PIT values are uniformly distributed under correct model specification, practical implementations rely on empirical estimates constructed from finite samples. We show that this estimation step fundamentally alters the statistical structure of the problem. In particular, common-sample and rolling-window implementations introduce dependence and variance distortions that invalidate classical one-sample uniformity tests. When empirical percentiles are conditioned on a shared reference sample, the resulting statistics converge towards a two-sample Kolmogorov--Smirnov regime, while rolling windows induce autocorrelation and variance suppression. Our findings indicate that treating empirical percentiles as independent uniform draws can distort statistical inference and that backtesting procedures based on PITs require revised calibration methods accounting for the underlying two-stage sampling structure.