Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimization

TL;DR

This paper investigates how Large Language Models manage exploration and exploitation in Bayesian optimization, proposing a multi-agent framework that improves search effectiveness by separating strategic control from candidate generation.

Contribution

It introduces a multi-agent approach that decomposes exploration-exploitation control into strategic and tactical components, enhancing LLM-based optimization performance.

Findings

Single-agent LLM approaches suffer from cognitive overload and unstable search dynamics.

Decomposing control into strategy and generation agents improves search effectiveness.

Empirical results show significant performance gains across optimization benchmarks.

Abstract

The exploration-exploitation trade-off is central to sequential decision-making and black-box optimization, yet how Large Language Models (LLMs) reason about and manage this trade-off remains poorly understood. Unlike Bayesian Optimization, where exploration and exploitation are explicitly encoded through acquisition functions, LLM-based optimization relies on implicit, prompt-based reasoning over historical evaluations, making search behavior difficult to analyze or control. In this work, we present a metric-level study of LLM-mediated search policy learning, studying how LLMs construct and adapt exploration-exploitation strategies under multiple operational definitions of exploration, including informativeness, diversity, and representativeness. We show that single-agent LLM approaches, which jointly perform strategy selection and candidate generation within a single prompt, suffer from cognitive overload, leading to unstable search dynamics and premature convergence. To address this limitation, we propose a multi-agent framework that decomposes exploration-exploitation control into strategic policy mediation and tactical candidate generation. A strategy agent assigns interpretable weights to multiple search criteria, while a generation agent produces candidates conditioned on the resulting search policy defined as weights. This decomposition renders exploration-exploitation decisions explicit, observable, and adjustable. Empirical results across various continuous optimization benchmarks indicate that separating strategic control from candidate generation substantially improves the effectiveness of LLM-mediated search.