Language Model Embeddings Can Be Sufficient for Bayesian Optimization

TL;DR

This paper demonstrates that language model embeddings of string inputs can serve as effective, flexible regressors in Bayesian Optimization, matching traditional methods in diverse optimization tasks.

Contribution

It introduces a novel approach using LLM embeddings for in-context regression in Bayesian Optimization, enabling general-purpose, domain-agnostic optimization.

Findings

Comparable optimization performance to Gaussian Process methods

Effective across synthetic, combinatorial, and hyperparameter domains

Shows potential for broader application and flexibility

Abstract

Bayesian Optimization is ubiquitous in experimental design and black-box optimization for improving search efficiency. However, most existing approaches rely on regression models which are limited to fixed search spaces and structured, tabular input features. This paper explores the use of LLM embeddings over string inputs for in-context regression in Bayesian Optimization. Our results show that representing inputs as strings enables general-purpose regression across diverse domains, including synthetic, combinatorial, and hyperparameter optimization. Furthermore, our approach achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier, and demonstrates potential for broader and more flexible applications.