Modeling Sampling Distributions of Test Statistics with Autograd

TL;DR

This paper investigates using autograd-enabled neural networks to model the sampling distribution of test statistics, offering an alternative to likelihood-ratio methods for simulation-based inference with scalar test statistics.

Contribution

It demonstrates that neural networks with autograd can effectively model the cdf of test statistics, providing a viable alternative to traditional likelihood-ratio approaches.

Findings

Neural network derivatives can approximate sampling distributions.

Autograd-based models perform comparably to likelihood-ratio methods.

Predictive uncertainty quantification enhances model reliability.

Abstract

Simulation-based inference methods that feature correct conditional coverage of confidence sets based on observations that have been compressed to a scalar test statistic require accurate modeling of either the p-value function or the cumulative distribution function (cdf) of the test statistic. If the model of the cdf, which is typically a deep neural network, is a function of the test statistic then the derivative of the neural network with respect to the test statistic furnishes an approximation of the sampling distribution of the test statistic. We explore whether this approach to modeling conditional 1-dimensional sampling distributions is a viable alternative to the probability density-ratio method, also known as the likelihood-ratio trick. Relatively simple, yet effective, neural network models are used whose predictive uncertainty is quantified through a variety of methods.