Measuring the robustness of Gaussian processes to kernel choice

TL;DR

This paper investigates how the choice of kernel in Gaussian processes affects decision-making in scientific and medical applications, revealing that decisions can be non-robust to different, qualitatively similar kernels.

Contribution

It introduces a method to quantify decision robustness across different kernels by solving a constrained optimization problem, highlighting potential non-robustness in practical scenarios.

Findings

Decisions with GPs can vary significantly with kernel choice.

Non-robustness persists even among qualitatively interchangeable kernels.

Method demonstrated on synthetic and real-world data.

Abstract

Gaussian processes (GPs) are used to make medical and scientific decisions, including in cardiac care and monitoring of atmospheric carbon dioxide levels. Notably, the choice of GP kernel is often somewhat arbitrary. In particular, uncountably many kernels typically align with qualitative prior knowledge (e.g.\ function smoothness or stationarity). But in practice, data analysts choose among a handful of convenient standard kernels (e.g.\ squared exponential). In the present work, we ask: Would decisions made with a GP differ under other, qualitatively interchangeable kernels? We show how to answer this question by solving a constrained optimization problem over a finite-dimensional space. We can then use standard optimizers to identify substantive changes in relevant decisions made with a GP. We demonstrate in both synthetic and real-world examples that decisions made with a GP can exhibit non-robustness to kernel choice, even when prior draws are qualitatively interchangeable to a user.