EigenGP: Sparse Gaussian process models with data-dependent eigenfunctions

TL;DR

EigenGP introduces a data-dependent sparse Gaussian process model using eigenfunctions derived from the Karhunen-Loeve expansion, enabling nonstationary covariance modeling and efficient semi-supervised learning.

Contribution

The paper proposes EigenGP, a novel sparse GP framework that employs Nystrom approximation for eigenfunctions, improving computational efficiency and nonstationary modeling capabilities.

Findings

Outperforms state-of-the-art sparse GP methods in regression and classification

Enables effective semi-supervised learning with both labeled and unlabeled data

Provides a scalable inference algorithm combining EP and EM techniques

Abstract

Gaussian processes (GPs) provide a nonparametric representation of functions. However, classical GP inference suffers from high computational cost and it is difficult to design nonstationary GP priors in practice. In this paper, we propose a sparse Gaussian process model, EigenGP, based on the Karhunen-Loeve (KL) expansion of a GP prior. We use the Nystrom approximation to obtain data dependent eigenfunctions and select these eigenfunctions by evidence maximization. This selection reduces the number of eigenfunctions in our model and provides a nonstationary covariance function. To handle nonlinear likelihoods, we develop an efficient expectation propagation (EP) inference algorithm, and couple it with expectation maximization for eigenfunction selection. Because the eigenfunctions of a Gaussian kernel are associated with clusters of samples - including both the labeled and unlabeled - selecting relevant eigenfunctions enables EigenGP to conduct semi-supervised learning. Our experimental results demonstrate improved predictive performance of EigenGP over alternative state-of-the-art sparse GP and semisupervised learning methods for regression, classification, and semisupervised classification.