Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings

TL;DR

This paper proposes leveraging language model embeddings to develop universal offline black-box optimization algorithms capable of handling heterogeneous data across multiple domains, overcoming traditional limitations.

Contribution

It introduces a novel approach using language model priors and embedding spaces to enable cross-domain universal black-box optimization.

Findings

Experiments show the proposed methods achieve universality and effectiveness.

Unifying language model priors with string embedding spaces overcomes traditional barriers.

The approach enables general-purpose black-box optimization across diverse data types.

Abstract

The pursuit of universal black-box optimization (BBO) algorithms is a longstanding goal. However, unlike domains such as language or vision, where scaling structured data has driven generalization, progress in offline BBO remains hindered by the lack of unified representations for heterogeneous numerical spaces. Thus, existing offline BBO approaches are constrained to single-task and fixed-dimensional settings, failing to achieve cross-domain universal optimization. Recent advances in language models (LMs) offer a promising path forward: their embeddings capture latent relationships in a unifying way, enabling universal optimization across different data types possible. In this paper, we discuss multiple potential approaches, including an end-to-end learning framework in the form of next-token prediction, as well as prioritizing the learning of latent spaces with strong representational capabilities. To validate the effectiveness of these methods, we collect offline BBO tasks and data from open-source academic works for training. Experiments demonstrate the universality and effectiveness of our proposed methods. Our findings suggest that unifying language model priors and learning string embedding space can overcome traditional barriers in universal BBO, paving the way for general-purpose BBO algorithms. The code is provided at https://github.com/lamda-bbo/universal-offline-bbo.