Physics Inspired Approaches To Understanding Gaussian Processes

TL;DR

This paper applies physics-inspired methods to analyze Gaussian Process models, revealing insights into their loss landscape, hyperparameter effects, and ensemble strategies, thereby enhancing understanding and practical performance of GPs.

Contribution

It introduces a physics-based analysis of GP loss landscapes, investigates hyperparameter optimization, and proposes methods for evaluating and improving GP ensembles.

Findings

$ u$-continuity demonstrated for Matern kernels

Optimal $ u$ values differ from typical literature choices

Physics-inspired ensemble evaluation methods improve GP performance

Abstract

Prior beliefs about the latent function to shape inductive biases can be incorporated into a Gaussian Process (GP) via the kernel. However, beyond kernel choices, the decision-making process of GP models remains poorly understood. In this work, we contribute an analysis of the loss landscape for GP models using methods from physics. We demonstrate $\nu$-continuity for Matern kernels and outline aspects of catastrophe theory at critical points in the loss landscape. By directly including $\nu$ in the hyperparameter optimisation for Matern kernels, we find that typical values of $\nu$ are far from optimal in terms of performance, yet prevail in the literature due to the increased computational speed. We also provide an a priori method for evaluating the effect of GP ensembles and discuss various voting approaches based on physical properties of the loss landscape. The utility of these approaches is demonstrated for various synthetic and real datasets. Our findings provide an enhanced understanding of the decision-making process behind GPs and offer practical guidance for improving their performance and interpretability in a range of applications.