Posterior Covariance Structures in Gaussian Processes

TL;DR

This paper provides a geometric analysis of the posterior covariance in Gaussian processes, showing how kernel parameters and data distribution influence covariance, and introduces estimators for efficient covariance measurement and approximation.

Contribution

It offers a novel geometric perspective on Gaussian process posterior covariance and proposes new estimators inspired by finite element methods for efficient covariance analysis.

Findings

Covariance magnitude varies with kernel bandwidth and data distribution.

Proposed estimators effectively measure and approximate posterior covariance.

Experiments validate theoretical insights and practical applications.

Abstract

In this paper, we present a comprehensive analysis of the posterior covariance field in Gaussian processes, with applications to the posterior covariance matrix. The analysis is based on the Gaussian prior covariance but the approach also applies to other covariance kernels. Our geometric analysis reveals how the Gaussian kernel's bandwidth parameter and the spatial distribution of the observations influence the posterior covariance as well as the corresponding covariance matrix, enabling straightforward identification of areas with high or low covariance in magnitude. Drawing inspiration from the a posteriori error estimation techniques in adaptive finite element methods, we also propose several estimators to efficiently measure the absolute posterior covariance field, which can be used for efficient covariance matrix approximation and preconditioning. We conduct a wide range of experiments to illustrate our theoretical findings and their practical applications.