Convergence of Gaussian Process Regression with Estimated Hyper-parameters and Applications in Bayesian Inverse Problems

TL;DR

This paper analyzes the convergence of hierarchical Gaussian process regression with estimated hyper-parameters, demonstrating that learning hyper-parameters from data does not hinder convergence and applying this to Bayesian inverse problems.

Contribution

It provides a convergence analysis for Gaussian process regression with hyper-parameter estimation, applicable to any continuous function, and assesses the impact on Bayesian inverse problem posteriors.

Findings

Convergence holds for any continuous function to be emulated.

Hyper-parameter learning does not affect the convergence guarantees.

Bound on the error in Bayesian posterior due to hyper-parameter estimation.

Abstract

This work is concerned with the convergence of Gaussian process regression. A particular focus is on hierarchical Gaussian process regression, where hyper-parameters appearing in the mean and covariance structure of the Gaussian process emulator are a-priori unknown, and are learnt from the data, along with the posterior mean and covariance. We work in the framework of empirical Bayes, where a point estimate of the hyper-parameters is computed, using the data, and then used within the standard Gaussian process prior to posterior update. We provide a convergence analysis that (i) holds for any continuous function $f$ to be emulated; and (ii) shows that convergence of Gaussian process regression is unaffected by the additional learning of hyper-parameters from data, and is guaranteed in a wide range of scenarios. As the primary motivation for the work is the use of Gaussian process regression to approximate the data likelihood in Bayesian inverse problems, we provide a bound on the error introduced in the Bayesian posterior distribution in this context.