FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch

TL;DR

FunBO leverages Large Language Models to automatically discover and learn new acquisition functions for Bayesian optimization, improving performance across diverse problems without manual tuning.

Contribution

This work introduces FunBO, an LLM-based approach to automatically discover and learn effective acquisition functions for Bayesian optimization, enhancing generalization and performance.

Findings

FunBO discovers acquisition functions that outperform standard ones.

The learned AFs generalize well across different optimization tasks.

FunBO achieves competitive results compared to specialized transfer-learning AFs.

Abstract

The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.