Dual Parameterization of Sparse Variational Gaussian Processes

TL;DR

This paper introduces a dual parameterization for sparse variational Gaussian processes that enhances computational efficiency and accuracy by utilizing dual parameters and natural gradient descent, without increasing memory usage.

Contribution

The paper proposes a novel dual parameterization for SVGPs that accelerates inference and improves hyperparameter learning accuracy.

Findings

Faster inference using natural gradient descent.

Tighter evidence lower bound for hyperparameter optimization.

Same memory cost as existing SVGP methods.

Abstract

Sparse variational Gaussian process (SVGP) methods are a common choice for non-conjugate Gaussian process inference because of their computational benefits. In this paper, we improve their computational efficiency by using a dual parameterization where each data example is assigned dual parameters, similarly to site parameters used in expectation propagation. Our dual parameterization speeds-up inference using natural gradient descent, and provides a tighter evidence lower bound for hyperparameter learning. The approach has the same memory cost as the current SVGP methods, but it is faster and more accurate.