Knot Selection in Sparse Gaussian Processes

TL;DR

This paper introduces a Bayesian optimization-based method for selecting knots in sparse Gaussian processes, improving accuracy and efficiency by avoiding issues with traditional marginal likelihood optimization.

Contribution

It proposes a novel one-at-a-time knot selection algorithm that adaptively determines the number and placement of knots using Bayesian optimization.

Findings

Enhanced accuracy over standard methods

Faster computation times

Reduced issues with multimodal likelihood surfaces

Abstract

Knot-based, sparse Gaussian processes have enjoyed considerable success as scalable approximations to full Gaussian processes. Problems can occur, however, when knot selection is done by optimizing the marginal likelihood. For example, the marginal likelihood surface is highly multimodal, which can cause suboptimal knot placement where some knots serve practically no function. This is especially a problem when many more knots are used than are necessary, resulting in extra computational cost for little to no gains in accuracy. We propose a one-at-a-time knot selection algorithm to select both the number and placement of knots. Our algorithm uses Bayesian optimization to efficiently propose knots that are likely to be good and largely avoids the pathologies encountered when using the marginal likelihood as the objective function. We provide empirical results showing improved accuracy and speed over the current standard approaches.