Amortising Inference and Meta-Learning Priors in Neural Networks

TL;DR

This paper proposes a novel method to learn priors for Bayesian neural networks using amortised variational inference, bridging Bayesian deep learning and meta-learning, enabling flexible priors and improved meta-learning capabilities.

Contribution

It introduces a neural process-based model that learns weight priors from datasets, allowing Bayesian neural networks to be used as generative models and to perform meta-learning with limited data.

Findings

Enables Bayesian neural networks to learn priors from data collections.

Allows meta-learning and generative modeling with Bayesian neural networks.

Supports within-task minibatching and data-starved meta-learning scenarios.

Abstract

One of the core facets of Bayesianism is in the updating of prior beliefs in light of new evidence$\text{ -- }$so how can we maintain a Bayesian approach if we have no prior beliefs in the first place? This is one of the central challenges in the field of Bayesian deep learning, where it is not clear how to represent beliefs about a prediction task by prior distributions over model parameters. Bridging the fields of Bayesian deep learning and probabilistic meta-learning, we introduce a way to $\textit{learn}$ a weights prior from a collection of datasets by introducing a way to perform per-dataset amortised variational inference. The model we develop can be viewed as a neural process whose latent variable is the set of weights of a BNN and whose decoder is the neural network parameterised by a sample of the latent variable itself. This unique model allows us to study the behaviour of Bayesian neural networks under well-specified priors, use Bayesian neural networks as flexible generative models, and perform desirable but previously elusive feats in neural processes such as within-task minibatching or meta-learning under extreme data-starvation.