Bayesian neural networks with interpretable priors from Mercer kernels

TL;DR

This paper introduces Mercer priors for Bayesian neural networks, enabling them to incorporate interpretable Gaussian process kernels, thus improving uncertainty quantification while maintaining scalability.

Contribution

It proposes a novel class of priors called Mercer priors that align BNNs with specified GPs without structural constraints, enhancing interpretability and scalability.

Findings

Mercer priors enable BNNs to approximate specified GPs.

The method improves uncertainty quantification in BNNs.

Scalable Bayesian inference is maintained with the new priors.

Abstract

Quantifying the uncertainty in the output of a neural network is essential for deployment in scientific or engineering applications where decisions must be made under limited or noisy data. Bayesian neural networks (BNNs) provide a framework for this purpose by constructing a Bayesian posterior distribution over the network parameters. However, the prior, which is of key importance in any Bayesian setting, is rarely meaningful for BNNs. This is because the complexity of the input-to-output map of a BNN makes it difficult to understand how certain distributions enforce any interpretable constraint on the output space of the network. Gaussian processes (GPs), on the other hand, are often preferred in uncertainty quantification tasks due to their interpretability. The drawback is that GPs are limited to small datasets without advanced techniques, which often rely on the covariance kernel having a specific structure. To address these challenges, we introduce a new class of priors for BNNs, called Mercer priors, such that the resulting BNN has samples which approximate that of a specified GP. The method works by defining a prior directly over the network parameters from the Mercer representation of the covariance kernel, and does not rely on the network having a specific structure. In doing so, we can exploit the scalability of BNNs in a meaningful Bayesian way.