Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks

TL;DR

This paper introduces a novel method leveraging neural tangent kernel theory to efficiently quantify and reduce uncertainty in over-parameterized neural networks, significantly lowering computational costs compared to traditional ensemble methods.

Contribution

It proposes a procedural-noise-correcting predictor that removes procedural uncertainty with only one auxiliary network, enabling efficient and reliable uncertainty quantification.

Findings

The PNC predictor effectively removes procedural uncertainty.

Constructs asymptotically exact confidence intervals with minimal networks.

Reduces computational effort compared to deep ensemble methods.

Abstract

Uncertainty quantification (UQ) is important for reliability assessment and enhancement of machine learning models. In deep learning, uncertainties arise not only from data, but also from the training procedure that often injects substantial noises and biases. These hinder the attainment of statistical guarantees and, moreover, impose computational challenges on UQ due to the need for repeated network retraining. Building upon the recent neural tangent kernel theory, we create statistically guaranteed schemes to principally \emph{characterize}, and \emph{remove}, the uncertainty of over-parameterized neural networks with very low computation effort. In particular, our approach, based on what we call a procedural-noise-correcting (PNC) predictor, removes the procedural uncertainty by using only \emph{one} auxiliary network that is trained on a suitably labeled dataset, instead of many retrained networks employed in deep ensembles. Moreover, by combining our PNC predictor with suitable light-computation resampling methods, we build several approaches to construct asymptotically exact-coverage confidence intervals using as low as four trained networks without additional overheads.