Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes

TL;DR

This paper introduces three improvements for iterative linear system solvers used in hyperparameter optimization of Gaussian processes, significantly speeding up computations and reducing residuals.

Contribution

It proposes a pathwise gradient estimator, warm starting, and early stopping techniques that enhance solver efficiency in large-scale Gaussian process hyperparameter optimization.

Findings

Speed-ups of up to 72x when solving to tolerance.

Residual norm decreased by up to 7x with early stopping.

Techniques are applicable across various linear system solvers.

Abstract

Scaling hyperparameter optimisation to very large datasets remains an open problem in the Gaussian process community. This paper focuses on iterative methods, which use linear system solvers, like conjugate gradients, alternating projections or stochastic gradient descent, to construct an estimate of the marginal likelihood gradient. We discuss three key improvements which are applicable across solvers: (i) a pathwise gradient estimator, which reduces the required number of solver iterations and amortises the computational cost of making predictions, (ii) warm starting linear system solvers with the solution from the previous step, which leads to faster solver convergence at the cost of negligible bias, (iii) early stopping linear system solvers after a limited computational budget, which synergises with warm starting, allowing solver progress to accumulate over multiple marginal likelihood steps. These techniques provide speed-ups of up to $72\times$ when solving to tolerance, and decrease the average residual norm by up to $7\times$ when stopping early.