Statistically Optimal Uncertainty Quantification for Expensive Black-Box Models

TL;DR

This paper develops a theoretical framework for constructing statistically optimal confidence intervals in the context of expensive black-box models, demonstrating that certain batching and resampling methods achieve minimal interval length under computational constraints.

Contribution

It introduces a new theoretical approach to identify the most efficient confidence interval methods for limited model evaluations, including novel formulas with uneven or overlapping batches.

Findings

Standard batching is optimal under constraints.

New batching formulas improve CI efficiency.

Resampling methods like cheap bootstrap are statistically optimal.

Abstract

Uncertainty quantification, by means of confidence interval (CI) construction, has been a fundamental problem in statistics and also important in risk-aware decision-making. In this paper, we revisit the basic problem of CI construction, but in the setting of expensive black-box models. This means we are confined to using a low number of model runs, and without the ability to obtain auxiliary model information such as gradients. In this case, there exist classical methods based on data splitting, and newer methods based on suitable resampling. However, while all these resulting CIs have similarly accurate coverage in large sample, their efficiencies in terms of interval length differ, and a systematic understanding of which method and configuration attains the shortest interval appears open. Motivated by this, we create a theoretical framework to study the statistical optimality on CI tightness under computation constraint. Our theory shows that standard batching, but also carefully constructed new formulas using uneven-size or overlapping batches, batched jackknife, and the so-called cheap bootstrap and its weighted generalizations, are statistically optimal. Our developments build on a new bridge of the classical notion of uniformly most accurate unbiasedness with batching and resampling, by viewing model runs as asymptotically Gaussian "data", as well as a suitable notion of homogeneity for CIs.