Shallow and Deep Nonparametric Convolutions for Gaussian Processes

TL;DR

This paper introduces a nonparametric process convolution approach for Gaussian processes that enhances flexibility, scalability, and the ability to infer covariance functions in deep models, outperforming standard methods on benchmarks.

Contribution

It proposes a novel nonparametric convolution framework for GPs, enabling scalable inference, multi-output modeling, and data-driven covariance learning in deep GP architectures.

Findings

Improves performance on large datasets compared to standard GPs.

Enables inference of covariance functions in deep GP models.

Supports multi-output Gaussian process modeling.

Abstract

A key challenge in the practical application of Gaussian processes (GPs) is selecting a proper covariance function. The moving average, or process convolutions, construction of GPs allows some additional flexibility, but still requires choosing a proper smoothing kernel, which is non-trivial. Previous approaches have built covariance functions by using GP priors over the smoothing kernel, and by extension the covariance, as a way to bypass the need to specify it in advance. However, such models have been limited in several ways: they are restricted to single dimensional inputs, e.g. time; they only allow modelling of single outputs and they do not scale to large datasets since inference is not straightforward. In this paper, we introduce a nonparametric process convolution formulation for GPs that alleviates these weaknesses by using a functional sampling approach based on Matheron's rule to perform fast sampling using interdomain inducing variables. Furthermore, we propose a composition of these nonparametric convolutions that serves as an alternative to classic deep GP models, and allows the covariance functions of the intermediate layers to be inferred from the data. We test the performance of our model on benchmarks for single output GPs, multiple output GPs and deep GPs and find that our approach can provide improvements over standard GP models, particularly for larger datasets.