In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics

TL;DR

This paper introduces LLaMEA-HPO, a hybrid framework that combines LLMs with hyper-parameter optimization to efficiently generate and tune heuristics, reducing computational costs while maintaining high performance.

Contribution

The paper presents a novel hybrid approach that integrates HPO into LLM-driven heuristic generation, significantly improving efficiency and reducing query costs.

Findings

LLaMEA-HPO outperforms existing frameworks on benchmark problems.

The hybrid approach reduces the number of LLM queries needed.

The method maintains high solution quality with lower computational costs.

Abstract

Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics, making them valuable tools in heuristic optimization tasks. However, LLMs are generally inefficient when it comes to fine-tuning hyper-parameters of the generated algorithms, often requiring excessive queries that lead to high computational and financial costs. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates the open source LLaMEA (Large Language Model Evolutionary Algorithm) framework with a Hyper-Parameter Optimization (HPO) procedure in the loop. By offloading hyper-parameter tuning to an HPO procedure, the LLaMEA-HPO framework allows the LLM to focus on generating novel algorithmic structures, reducing the number of required LLM queries and improving the overall efficiency of the optimization process. We empirically validate the proposed hybrid framework on benchmark problems, including Online Bin Packing, Black-Box Optimization, and the Traveling Salesperson Problem. Our results demonstrate that LLaMEA-HPO achieves superior or comparable performance compared to existing LLM-driven frameworks while significantly reducing computational costs. This work highlights the importance of separating algorithmic innovation and structural code search from parameter tuning in LLM-driven code optimization and offers a scalable approach to improve the efficiency and effectiveness of LLM-based code generation.