Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data

TL;DR

This paper introduces iterative methods with novel preconditioners to improve the computational efficiency of full-scale Gaussian process approximations for large spatial datasets, enabling faster likelihood, gradient, and predictive variance calculations.

Contribution

It proposes new preconditioners and iterative algorithms that significantly accelerate Gaussian process approximations, outperforming existing methods in speed and accuracy.

Findings

Accelerated convergence of conjugate gradient method with new preconditioner.

Outperforms state-of-the-art pivoted Cholesky preconditioner.

Efficient calculation of predictive variances using stochastic simulation.

Abstract

Gaussian processes are flexible probabilistic regression models which are widely used in statistics and machine learning. However, a drawback is their limited scalability to large data sets. To alleviate this, full-scale approximations (FSAs) combine predictive process methods and covariance tapering, thus approximating both global and local structures. We show how iterative methods can be used to reduce computational costs in calculating likelihoods, gradients, and predictive distributions with FSAs. In particular, we introduce a novel preconditioner and show theoretically and empirically that it accelerates the conjugate gradient method's convergence speed and mitigates its sensitivity with respect to the FSA parameters and the eigenvalue structure of the original covariance matrix, and we demonstrate empirically that it outperforms a state-of-the-art pivoted Cholesky preconditioner. Furthermore, we introduce an accurate and fast way to calculate predictive variances using stochastic simulation and iterative methods. In addition, we show how our newly proposed fully independent training conditional (FITC) preconditioner can also be used in iterative methods for Vecchia approximations. In our experiments, it outperforms existing state-of-the-art preconditioners for Vecchia approximations. All methods are implemented in a free C++ software library with high-level Python and R packages.