Robust Gaussian Processes via Relevance Pursuit

TL;DR

This paper introduces a robust Gaussian process model that infers data-specific noise levels to handle outliers, leveraging a novel relevance pursuit method with strong theoretical guarantees, improving robustness in regression and Bayesian optimization tasks.

Contribution

It proposes a new GP model with a relevance pursuit algorithm that infers data-specific noise levels, offering strong concavity and approximation guarantees for robustness against outliers.

Findings

Model outperforms existing methods on diverse tasks

Strong concavity of the marginal likelihood in noise variances

Approximation guarantees due to weak submodularity

Abstract

Gaussian processes (GPs) are non-parametric probabilistic regression models that are popular due to their flexibility, data efficiency, and well-calibrated uncertainty estimates. However, standard GP models assume homoskedastic Gaussian noise, while many real-world applications are subject to non-Gaussian corruptions. Variants of GPs that are more robust to alternative noise models have been proposed, and entail significant trade-offs between accuracy and robustness, and between computational requirements and theoretical guarantees. In this work, we propose and study a GP model that achieves robustness against sparse outliers by inferring data-point-specific noise levels with a sequential selection procedure maximizing the log marginal likelihood that we refer to as relevance pursuit. We show, surprisingly, that the model can be parameterized such that the associated log marginal likelihood is strongly concave in the data-point-specific noise variances, a property rarely found in either robust regression objectives or GP marginal likelihoods. This in turn implies the weak submodularity of the corresponding subset selection problem, and thereby proves approximation guarantees for the proposed algorithm. We compare the model's performance relative to other approaches on diverse regression and Bayesian optimization tasks, including the challenging but common setting of sparse corruptions of the labels within or close to the function range.