Predictive Complexity Priors

TL;DR

This paper introduces predictive complexity priors, a new type of Bayesian prior based on model predictions, which simplifies prior specification for complex models like neural networks and improves various applications.

Contribution

It proposes predictive complexity priors that are defined through model predictions and transferred to parameters, facilitating Bayesian inference in complex models.

Findings

Effective in high-dimensional regression

Improves neural network depth modeling

Enhances few-shot learning performance

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.