Understanding Probabilistic Sparse Gaussian Process Approximations

TL;DR

This paper compares two popular sparse Gaussian Process approximation methods, FITC and VFE, analyzing their theoretical differences and practical behaviors to guide better application in large-scale regression tasks.

Contribution

It provides a thorough analytical and empirical comparison of FITC and VFE, clarifying their differences and practical implications for Gaussian Process regression.

Findings

FITC and VFE have different theoretical properties.

The two methods behave differently in practice.

Guidelines for choosing between FITC and VFE.

Abstract

Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.