Evolve Cost-aware Acquisition Functions Using Large Language Models

TL;DR

EvolCAF leverages large language models and evolutionary algorithms to automatically design cost-aware acquisition functions for Bayesian optimization, reducing manual effort and improving efficiency across diverse optimization tasks.

Contribution

This work introduces a novel framework combining LLMs and evolutionary computation to automate the design of cost-aware acquisition functions, enhancing generalization and interpretability.

Findings

Outperforms human-designed methods like EIpu and EI-cool in efficiency.

Demonstrates strong generalization across synthetic and real-world tasks.

Reduces reliance on domain expertise and manual trial-and-error.

Abstract

Many real-world optimization scenarios involve expensive evaluation with unknown and heterogeneous costs. Cost-aware Bayesian optimization stands out as a prominent solution in addressing these challenges. To approach the global optimum within a limited budget in a cost-efficient manner, the design of cost-aware acquisition functions (AFs) becomes a crucial step. However, traditional manual design paradigm typically requires extensive domain knowledge and involves a labor-intensive trial-and-error process. This paper introduces EvolCAF, a novel framework that integrates large language models (LLMs) with evolutionary computation (EC) to automatically design cost-aware AFs. Leveraging the crossover and mutation in the algorithmic space, EvolCAF offers a novel design paradigm, significantly reduces the reliance on domain expertise and model training. The designed cost-aware AF maximizes the utilization of available information from historical data, surrogate models and budget details. It introduces novel ideas not previously explored in the existing literature on acquisition function design, allowing for clear interpretations to provide insights into its behavior and decision-making process. In comparison to the well-known EIpu and EI-cool methods designed by human experts, our approach showcases remarkable efficiency and generalization across various tasks, including 12 synthetic problems and 3 real-world hyperparameter tuning test sets.