Variational sparse inverse Cholesky approximation for latent Gaussian processes via double Kullback-Leibler minimization

TL;DR

This paper introduces a scalable variational method for latent Gaussian processes using sparse inverse Cholesky factors, achieving high accuracy with efficient computation, especially for stationary kernels.

Contribution

It develops a novel double-Kullback-Leibler minimization approach with sparse inverse Cholesky structures, improving inference accuracy and computational efficiency for Gaussian processes.

Findings

Outperforms inducing-point methods in accuracy for stationary kernels

Achieves polylogarithmic time per iteration in stochastic gradient descent

Provides highly accurate prior and posterior approximations with sparsity patterns

Abstract

To achieve scalable and accurate inference for latent Gaussian processes, we propose a variational approximation based on a family of Gaussian distributions whose covariance matrices have sparse inverse Cholesky (SIC) factors. We combine this variational approximation of the posterior with a similar and efficient SIC-restricted Kullback-Leibler-optimal approximation of the prior. We then focus on a particular SIC ordering and nearest-neighbor-based sparsity pattern resulting in highly accurate prior and posterior approximations. For this setting, our variational approximation can be computed via stochastic gradient descent in polylogarithmic time per iteration. We provide numerical comparisons showing that the proposed double-Kullback-Leibler-optimal Gaussian-process approximation (DKLGP) can sometimes be vastly more accurate for stationary kernels than alternative approaches such as inducing-point and mean-field approximations at similar computational complexity.