A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian Processes

TL;DR

This paper unifies various non-stationary kernels for Gaussian processes, demonstrating their properties and performance, and introduces a new kernel that leverages their advantages for improved prediction and uncertainty quantification.

Contribution

It provides a comprehensive perspective on non-stationary kernels, compares their effectiveness, and proposes a novel kernel that combines their beneficial features.

Findings

Non-stationary kernels outperform stationary ones in certain scenarios.

The proposed kernel shows improved prediction accuracy.

Different non-stationary kernels have distinct advantages depending on data characteristics.

Abstract

The Gaussian process (GP) is a popular statistical technique for stochastic function approximation and uncertainty quantification from data. GPs have been adopted into the realm of machine learning in the last two decades because of their superior prediction abilities, especially in data-sparse scenarios, and their inherent ability to provide robust uncertainty estimates. Even so, their performance highly depends on intricate customizations of the core methodology, which often leads to dissatisfaction among practitioners when standard setups and off-the-shelf software tools are being deployed. Arguably the most important building block of a GP is the kernel function which assumes the role of a covariance operator. Stationary kernels of the Mat\'ern class are used in the vast majority of applied studies; poor prediction performance and unrealistic uncertainty quantification are often the consequences. Non-stationary kernels show improved performance but are rarely used due to their more complicated functional form and the associated effort and expertise needed to define and tune them optimally. In this perspective, we want to help ML practitioners make sense of some of the most common forms of non-stationarity for Gaussian processes. We show a variety of kernels in action using representative datasets, carefully study their properties, and compare their performances. Based on our findings, we propose a new kernel that combines some of the identified advantages of existing kernels.