Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models

TL;DR

This paper evaluates the role of evolutionary search in automated heuristic design using large language models, demonstrating its importance through extensive benchmarking and analysis.

Contribution

It provides a comprehensive benchmark and analysis showing the significance of evolutionary search in LLM-based automated heuristic design, addressing prior gaps.

Findings

Evolutionary search is crucial for effective LLM-based heuristic design.

Benchmark results highlight the impact of search strategies on performance.

Open-sourced dataset and results facilitate future research.

Abstract

Automated heuristic design (AHD) has gained considerable attention for its potential to automate the development of effective heuristics. The recent advent of large language models (LLMs) has paved a new avenue for AHD, with initial efforts focusing on framing AHD as an evolutionary program search (EPS) problem. However, inconsistent benchmark settings, inadequate baselines, and a lack of detailed component analysis have left the necessity of integrating LLMs with search strategies and the true progress achieved by existing LLM-based EPS methods to be inadequately justified. This work seeks to fulfill these research queries by conducting a large-scale benchmark comprising four LLM-based EPS methods and four AHD problems across nine LLMs and five independent runs. Our extensive experiments yield meaningful insights, providing empirical grounding for the importance of evolutionary search in LLM-based AHD approaches, while also contributing to the advancement of future EPS algorithmic development. To foster accessibility and reproducibility, we have fully open-sourced our benchmark and corresponding results.