An Overview of Uncertainty Quantification Methods for Infinite Neural Networks

TL;DR

This paper reviews methods for quantifying uncertainty in infinite-width neural networks, highlighting their connections to Gaussian processes and providing exact solutions for predictive uncertainty.

Contribution

It offers a comprehensive overview of uncertainty quantification techniques for infinite neural networks and clarifies their relationship with Gaussian process models.

Findings

Exact closed-form solutions for predictive uncertainty.

Comparison of uncertainty methods with Gaussian processes.

Insights into the Bayesian inference framework for infinite networks.

Abstract

To better understand the theoretical behavior of large neural networks, several works have analyzed the case where a network's width tends to infinity. In this regime, the effect of random initialization and the process of training a neural network can be formally expressed with analytical tools like Gaussian processes and neural tangent kernels. In this paper, we review methods for quantifying uncertainty in such infinite-width neural networks and compare their relationship to Gaussian processes in the Bayesian inference framework. We make use of several equivalence results along the way to obtain exact closed-form solutions for predictive uncertainty.