Variational Orthogonal Features

TL;DR

This paper introduces a new class of variational features for stationary kernels that enable more efficient Gaussian process inference by reducing computational costs while maintaining accuracy.

Contribution

It proposes a construction of variational features applicable to any stationary kernel, significantly reducing inference complexity in Gaussian processes.

Findings

Unbiased ELBO estimator computed in ( ilde{N}T+M^2T) time

Approximate estimator computed in ( ilde{N}T+MT) time

Analysis of approximation impact on inference quality

Abstract

Sparse stochastic variational inference allows Gaussian process models to be applied to large datasets. The per iteration computational cost of inference with this method is $\mathcal{O}(\tilde{N}M^2+M^3),$ where $\tilde{N}$ is the number of points in a minibatch and $M$ is the number of `inducing features', which determine the expressiveness of the variational family. Several recent works have shown that for certain priors, features can be defined that remove the $\mathcal{O}(M^3)$ cost of computing a minibatch estimate of an evidence lower bound (ELBO). This represents a significant computational savings when $M\gg \tilde{N}$. We present a construction of features for any stationary prior kernel that allow for computation of an unbiased estimator to the ELBO using $T$ Monte Carlo samples in $\mathcal{O}(\tilde{N}T+M^2T)$ and in $\mathcal{O}(\tilde{N}T+MT)$ with an additional approximation. We analyze the impact of this additional approximation on inference quality.