Exact posterior distributions of wide Bayesian neural networks

TL;DR

This paper proves that the exact Bayesian neural network (BNN) posterior converges to the Gaussian process limit as the network width increases, and demonstrates how to generate exact BNN samples on small datasets.

Contribution

It provides the first theoretical proof of the convergence of the exact BNN posterior to the GP limit and introduces a method for exact sampling from finite BNNs.

Findings

Exact BNN posterior converges to GP limit as width increases

Method for generating exact BNN samples on small datasets

Theoretical validation of BNN posterior convergence

Abstract

Recent work has shown that the prior over functions induced by a deep Bayesian neural network (BNN) behaves as a Gaussian process (GP) as the width of all layers becomes large. However, many BNN applications are concerned with the BNN function space posterior. While some empirical evidence of the posterior convergence was provided in the original works of Neal (1996) and Matthews et al. (2018), it is limited to small datasets or architectures due to the notorious difficulty of obtaining and verifying exactness of BNN posterior approximations. We provide the missing theoretical proof that the exact BNN posterior converges (weakly) to the one induced by the GP limit of the prior. For empirical validation, we show how to generate exact samples from a finite BNN on a small dataset via rejection sampling.