Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited

TL;DR

This paper introduces a block-diagonal structured approximation for sparse Gaussian processes, improving variational bounds and leveraging Power Expectation Propagation for better performance and flexibility.

Contribution

It extends sparse GP methods with block-diagonal scaling matrices and revisits Power Expectation Propagation to incorporate these structured approximations.

Findings

Block-diagonal approximation tightens variational lower bounds.

Structured approximations perform as well or better than diagonal ones.

PEP framework with structured posteriors offers flexible, competitive results.

Abstract

Inducing-point-based sparse variational Gaussian processes have become the standard workhorse for scaling up GP models. Recent advances show that these methods can be improved by introducing a diagonal scaling matrix to the conditional posterior density given the inducing points. This paper first considers an extension that employs a block-diagonal structure for the scaling matrix, provably tightening the variational lower bound. We then revisit the unifying framework of sparse GPs based on Power Expectation Propagation (PEP) and show that it can leverage and benefit from the new structured approximate posteriors. Through extensive regression experiments, we show that the proposed block-diagonal approximation consistently performs similarly to or better than existing diagonal approximations while maintaining comparable computational costs. Furthermore, the new PEP framework with structured posteriors provides competitive performance across various power hyperparameter settings, offering practitioners flexible alternatives to standard variational approaches.