Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds

TL;DR

This paper introduces a novel method for training Gaussian Processes by directly optimizing PAC-Bayesian bounds, leading to improved generalization guarantees and robustness, especially relevant for safety-critical applications.

Contribution

It proposes a new learning approach for GPs that focuses on PAC-Bayesian bounds rather than marginal likelihood, enhancing theoretical guarantees and practical robustness.

Findings

Better generalization guarantees than traditional methods

Robustness across multiple regression benchmarks

Significant improvements in safety-critical scenarios

Abstract

Gaussian Processes (GPs) are a generic modelling tool for supervised learning. While they have been successfully applied on large datasets, their use in safety-critical applications is hindered by the lack of good performance guarantees. To this end, we propose a method to learn GPs and their sparse approximations by directly optimizing a PAC-Bayesian bound on their generalization performance, instead of maximizing the marginal likelihood. Besides its theoretical appeal, we find in our evaluation that our learning method is robust and yields significantly better generalization guarantees than other common GP approaches on several regression benchmark datasets.