Probabilistic Programming with Gaussian Process Memoization

TL;DR

This paper introduces gpmem, a flexible probabilistic programming interface for Gaussian processes that simplifies complex applications like regression, symbolic discovery, and Bayesian optimization, demonstrated through three diverse case studies.

Contribution

It presents a novel memoization-based approach to embed Gaussian processes in probabilistic programming languages, enabling easy implementation of advanced GP applications.

Findings

gpmem improves regression with hierarchical hyper-parameter learning

enables Bayesian structure learning for symbolic expressions from data

facilitates Bayesian optimization with automatic inference and action selection

Abstract

Gaussian Processes (GPs) are widely used tools in statistics, machine learning, robotics, computer vision, and scientific computation. However, despite their popularity, they can be difficult to apply; all but the simplest classification or regression applications require specification and inference over complex covariance functions that do not admit simple analytical posteriors. This paper shows how to embed Gaussian processes in any higher-order probabilistic programming language, using an idiom based on memoization, and demonstrates its utility by implementing and extending classic and state-of-the-art GP applications. The interface to Gaussian processes, called gpmem, takes an arbitrary real-valued computational process as input and returns a statistical emulator that automatically improve as the original process is invoked and its input-output behavior is recorded. The flexibility of gpmem is illustrated via three applications: (i) robust GP regression with hierarchical hyper-parameter learning, (ii) discovering symbolic expressions from time-series data by fully Bayesian structure learning over kernels generated by a stochastic grammar, and (iii) a bandit formulation of Bayesian optimization with automatic inference and action selection. All applications share a single 50-line Python library and require fewer than 20 lines of probabilistic code each.