Towards Improved Learning in Gaussian Processes: The Best of Two Worlds

TL;DR

This paper proposes a hybrid training method for Gaussian processes that combines the strengths of Expectation Propagation and Variational Inference, leading to improved hyperparameter learning and better generalization in binary classification.

Contribution

It introduces a novel hybrid inference and learning procedure that leverages conjugate-computation VI and EP-like marginal likelihood approximation for Gaussian processes.

Findings

Hybrid method improves hyperparameter learning.

Method generalizes better in binary classification.

Empirical results demonstrate superior performance.

Abstract

Gaussian process training decomposes into inference of the (approximate) posterior and learning of the hyperparameters. For non-Gaussian (non-conjugate) likelihoods, two common choices for approximate inference are Expectation Propagation (EP) and Variational Inference (VI), which have complementary strengths and weaknesses. While VI's lower bound to the marginal likelihood is a suitable objective for inferring the approximate posterior, it does not automatically imply it is a good learning objective for hyperparameter optimization. We design a hybrid training procedure where the inference leverages conjugate-computation VI and the learning uses an EP-like marginal likelihood approximation. We empirically demonstrate on binary classification that this provides a good learning objective and generalizes better.