SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization

TL;DR

SemanticOpt leverages large language models with semantic domain knowledge to improve black-box optimization, outperforming classical and existing LLM-based methods on diverse real-world problems.

Contribution

Introduces SemanticOpt, a novel framework that fine-tunes LLMs with structured Bayesian optimization trajectories and semantic context for enhanced black-box optimization.

Findings

SemanticOpt outperforms classical optimizers on real-world tasks.

SemanticOpt surpasses existing LLM-based approaches in diverse benchmarks.

The framework effectively integrates semantic information into optimization.

Abstract

Optimizing an experimental system can be extremely challenging when each experiment is expensive, time-consuming, or difficult to perform. Existing optimizers for expensive black-box problems, such as Bayesian optimization, are typically limited to numerical or categorical observations. They do not make use of broader domain knowledge, such as expert heuristics, relevant scientific papers, or similar previous experiments. Large language models (LLMs) can interpret this semantic information; however, even state-of-the-art LLMs struggle to reliably solve black-box optimization problems. We introduce SemanticOpt, a framework for semantic black-box optimization that equips LLMs with optimization capabilities by fine-tuning them on structured Bayesian optimization trajectories augmented with natural-language context. SemanticOpt jointly uses numerical and semantic evidence when proposing new experiments, while producing interpretable predictions aligned with Bayesian surrogate models. We construct a range of real-world optimization problems paired with semantic information to create a diverse benchmark for evaluating semantic black-box optimization. Across these domains, SemanticOpt outperforms both classical optimizers and existing LLM-based approaches on average when given relevant semantic information.