The Implicit Delta Method

TL;DR

This paper introduces the implicit delta method, a novel approach for quantifying epistemic uncertainty in complex predictive models by infinitesimally regularizing the training loss, enabling reliable standard errors and confidence intervals.

Contribution

The paper proposes the implicit delta method, an innovative technique that assesses uncertainty through infinitesimal regularization, avoiding computationally intensive procedures like bootstrapping.

Findings

The implicit delta method provides consistent estimates of asymptotic variance.

It enables construction of calibrated confidence intervals.

Empirical results demonstrate its effectiveness in uncertainty quantification.

Abstract

Epistemic uncertainty quantification is a crucial part of drawing credible conclusions from predictive models, whether concerned about the prediction at a given point or any downstream evaluation that uses the model as input. When the predictive model is simple and its evaluation differentiable, this task is solved by the delta method, where we propagate the asymptotically-normal uncertainty in the predictive model through the evaluation to compute standard errors and Wald confidence intervals. However, this becomes difficult when the model and/or evaluation becomes more complex. Remedies include the bootstrap, but it can be computationally infeasible when training the model even once is costly. In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of the predictive model to automatically assess downstream uncertainty. We show that the change in the evaluation due to regularization is consistent for the asymptotic variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference. This provides both a reliable quantification of uncertainty in terms of standard errors as well as permits the construction of calibrated confidence intervals. We discuss connections to other approaches to uncertainty quantification, both Bayesian and frequentist, and demonstrate our approach empirically.