Scaling Gaussian Process Regression with Derivatives

TL;DR

This paper introduces efficient iterative methods for Gaussian process regression with derivatives, significantly reducing computational costs and enabling high-dimensional Bayesian optimization with derivatives.

Contribution

It proposes fast iterative solvers with pivoted Cholesky preconditioning for scalable GP regression with derivatives, improving efficiency over traditional methods.

Findings

Iterative solvers with MVMs reduce computation time.

Pivoted Cholesky preconditioning accelerates convergence.

Enables high-dimensional Bayesian optimization with derivatives.

Abstract

Gaussian processes (GPs) with derivatives are useful in many applications, including Bayesian optimization, implicit surface reconstruction, and terrain reconstruction. Fitting a GP to function values and derivatives at $n$ points in $d$ dimensions requires linear solves and log determinants with an ${n(d+1) \times n(d+1)}$ positive definite matrix -- leading to prohibitive $\mathcal{O}(n^3d^3)$ computations for standard direct methods. We propose iterative solvers using fast $\mathcal{O}(nd)$ matrix-vector multiplications (MVMs), together with pivoted Cholesky preconditioning that cuts the iterations to convergence by several orders of magnitude, allowing for fast kernel learning and prediction. Our approaches, together with dimensionality reduction, enables Bayesian optimization with derivatives to scale to high-dimensional problems and large evaluation budgets.