Sequential Subspace Search for Functional Bayesian Optimization Incorporating Experimenter Intuition

TL;DR

This paper introduces a Bayesian functional optimization algorithm that incorporates experimenter intuition through Gaussian Process priors, iteratively exploring subspaces to efficiently find optimal functions with proven convergence guarantees.

Contribution

The paper presents a novel sequential subspace search algorithm for Bayesian functional optimization that leverages experimenter beliefs and provides theoretical convergence analysis.

Findings

Algorithm converges in sub-linear time with finite effective dimension.

Successfully applied to simulated and real-world problems.

Provides regret bounds for the optimization process.

Abstract

We propose an algorithm for Bayesian functional optimisation - that is, finding the function to optimise a process - guided by experimenter beliefs and intuitions regarding the expected characteristics (length-scale, smoothness, cyclicity etc.) of the optimal solution encoded into the covariance function of a Gaussian Process. Our algorithm generates a sequence of finite-dimensional random subspaces of functional space spanned by a set of draws from the experimenter's Gaussian Process. Standard Bayesian optimisation is applied on each subspace, and the best solution found used as a starting point (origin) for the next subspace. Using the concept of effective dimensionality, we analyse the convergence of our algorithm and provide a regret bound to show that our algorithm converges in sub-linear time provided a finite effective dimension exists. We test our algorithm in simulated and real-world experiments, namely blind function matching, finding the optimal precipitation-strengthening function for an aluminium alloy, and learning rate schedule optimisation for deep networks.