Evolution of Benchmark: Black-Box Optimization Benchmark Design through Large Language Model

TL;DR

This paper introduces EoB, an automated benchmark design method for black-box optimization that uses large language models to evolve diverse and effective benchmarks, reducing human bias and increasing diversity.

Contribution

We propose EoB, a novel LLM-powered framework that automates BBO benchmark creation through bi-objective optimization and program evolution, enhancing diversity and objectivity.

Findings

EoB produces competitive benchmarks for algorithm evaluation.

EoB aids in training and testing learning-assisted BBO algorithms.

EoB extends to proxies for expensive real-world problems.

Abstract

Benchmark Design in Black-Box Optimization (BBO) is a fundamental yet open-ended topic. Early BBO benchmarks are predominantly human-crafted, introducing expert bias and constraining diversity. Automating this design process can relieve the human-in-the-loop burden while enhancing diversity and objectivity. We propose Evolution of Benchmark (EoB), an automated BBO benchmark designer empowered by the large language model (LLM) and its program evolution capability. Specifically, we formulate benchmark design as a bi-objective optimization problem towards maximizing (i) landscape diversity and (ii) algorithm-differentiation ability across a portfolio of BBO solvers. Under this paradigm, EoB iteratively prompts LLM to evolve a population of benchmark programs and employs a reflection-based scheme to co-evolve the landscape and its corresponding program. Comprehensive experiments validate our EoB is a competitive candidate in multi-dimensional usages: 1) Benchmarking BBO algorithms; 2) Training and testing learning-assisted BBO algorithms; 3) Extending proxy for expensive real-world problems.