Approximate Inference for Fully Bayesian Gaussian Process Regression

TL;DR

This paper explores approximate Bayesian inference methods for hyperparameters in Gaussian Process Regression, comparing sampling and variational approaches to improve predictive performance.

Contribution

It introduces and analyzes two approximation schemes—HMC and Variational Inference—for the hyperparameter posterior in fully Bayesian GPR.

Findings

HMC provides accurate posterior samples for hyperparameters.

Variational Inference offers a computationally efficient approximation.

Fully Bayesian GPR improves predictive performance on benchmark datasets.

Abstract

Learning in Gaussian Process models occurs through the adaptation of hyperparameters of the mean and the covariance function. The classical approach entails maximizing the marginal likelihood yielding fixed point estimates (an approach called \textit{Type II maximum likelihood} or ML-II). An alternative learning procedure is to infer the posterior over hyperparameters in a hierarchical specification of GPs we call \textit{Fully Bayesian Gaussian Process Regression} (GPR). This work considers two approximation schemes for the intractable hyperparameter posterior: 1) Hamiltonian Monte Carlo (HMC) yielding a sampling-based approximation and 2) Variational Inference (VI) where the posterior over hyperparameters is approximated by a factorized Gaussian (mean-field) or a full-rank Gaussian accounting for correlations between hyperparameters. We analyze the predictive performance for fully Bayesian GPR on a range of benchmark data sets.