An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models

TL;DR

This paper introduces an evolutionary framework that uses large language models to automatically generate diverse and challenging optimization benchmarks, enabling better assessment of algorithm performance.

Contribution

It presents a novel LLM-driven evolutionary method for creating optimization benchmarks that reflect real-world problem diversity and complexity.

Findings

Generated benchmarks show significant performance differences between algorithms.

Over 80% of generated problems favor the target algorithm.

Benchmarks exhibit distinct geometric characteristics affecting algorithm performance.

Abstract

Optimization benchmarks play a fundamental role in assessing algorithm performance; however, existing artificial benchmarks often fail to capture the diversity and irregularity of real-world problem structures, while benchmarks derived from real-world problems are costly and difficult to construct. To address these challenges, we propose an evolutionary automatic benchmark generation framework that leverages a large language model (LLM) as a generative operator, termed the LLM-driven evolutionary benchmark generator (LLM-EBG). In this framework, the LLM serves as an evolutionary operator that generates and evolves benchmark problems within a flexible, expressive representation space. As a case study, we generate unconstrained single-objective continuous minimization problems represented as mathematical expressions designed to induce significant performance differences between a genetic algorithm (GA) and differential evolution (DE). Experimental results show that LLM-EBG successfully produces benchmark problems in which the designated target algorithm consistently outperforms the comparative algorithm in more than 80\% of trials. Furthermore, exploratory landscape analysis reveals that benchmarks favoring GA are highly sensitive to variable scaling, demonstrating that the proposed framework can generate problems with distinct geometric characteristics that reflect the intrinsic search behaviors of different optimization algorithms.