Design-marginal calibration of Gaussian process predictive distributions: Bayesian and conformal approaches

TL;DR

This paper introduces two novel methods for calibrating Gaussian process predictive distributions using Bayesian and conformal approaches, improving coverage and calibration in sequential design settings.

Contribution

It formalizes design-marginal calibration for GPs and proposes cps-gp and bcr-gp methods that enhance predictive calibration and distribution smoothness.

Findings

cps-gp achieves finite-sample marginal calibration

bcr-gp controls dispersion and tail behavior effectively

methods outperform benchmarks in calibration metrics

Abstract

We study the calibration of Gaussian process (GP) predictive distributions in the interpolation setting from a design-marginal perspective. Conditioning on the data and averaging over a design measure \mu, we formalize \mu-coverage for central intervals and \mu-probabilistic calibration through randomized probability integral transforms. We introduce two methods. cps-gp adapts conformal predictive systems to GP interpolation using standardized leave-one-out residuals, yielding stepwise predictive distributions with finite-sample marginal calibration. bcr-gp retains the GP posterior mean and replaces the Gaussian residual by a generalized normal model fitted to cross-validated standardized residuals. A Bayesian selection rule-based either on a posterior upper quantile of the variance for conservative prediction or on a cross-posterior Kolmogorov-Smirnov criterion for probabilistic calibration-controls dispersion and tail behavior while producing smooth predictive distributions suitable for sequential design. Numerical experiments on benchmark functions compare cps-gp, bcr-gp, Jackknife+ for GPs, and the full conformal Gaussian process, using calibration metrics (coverage, Kolmogorov-Smirnov, integral absolute error) and accuracy or sharpness through the scaled continuous ranked probability score.