Warped Gradient-Enhanced Gaussian Process Surrogate Models for Exponential Family Likelihoods with Intractable Normalizing Constants

TL;DR

This paper introduces a warped, gradient-enhanced Gaussian process surrogate model for exponential family likelihoods with intractable normalizing constants, improving approximation accuracy and computational efficiency in Bayesian inference.

Contribution

It proposes a novel surrogate model that captures nonstationarity and leverages gradient information, outperforming traditional Gaussian process models in complex likelihood scenarios.

Findings

Outperforms stationary Gaussian process surrogates in phase transition regions

Reduces computational cost of MCMC inference for intractable likelihoods

Enhances effective sample size per unit time in Bayesian algorithms

Abstract

Markov chain Monte Carlo methods for exponential family models with intractable normalizing constant, such as the exchange algorithm, require simulations of the sufficient statistics at every iteration of the Markov chain, which often result in expensive computations. Surrogate models for the likelihood function have been developed to accelerate inference algorithms in this context. However, these surrogate models tend to be relatively inflexible, and often provide a poor approximation to the true likelihood function. In this article, we propose the use of a warped, gradient-enhanced, Gaussian process surrogate model for the likelihood function, which jointly models the sample means and variances of the sufficient statistics, and uses warping functions to capture covariance nonstationarity in the input parameter space. We show that both the consideration of nonstationarity and the inclusion of gradient information can be leveraged to obtain a surrogate model that outperforms the conventional stationary Gaussian process surrogate model when making inference, particularly in regions where the likelihood function exhibits a phase transition. We also show that the proposed surrogate model can be used to improve the effective sample size per unit time when embedded in exact inferential algorithms. The utility of our approach in speeding up inferential algorithms is demonstrated on simulated and real-world data.