Advancing Automated Algorithm Design via Evolutionary Stagewise Design with LLMs

TL;DR

EvoStage introduces a stagewise, feedback-driven evolutionary framework utilizing LLMs for automated algorithm design, significantly outperforming existing methods in chip placement and optimization tasks.

Contribution

The paper presents EvoStage, a novel multi-stage evolutionary approach with real-time feedback and multi-agent systems, enhancing LLM-based algorithm design for industrial applications.

Findings

Outperforms human-expert and existing LLM methods in benchmarks.

Achieves state-of-the-art wire-length results on chip placement.

Surpasses original metrics in a commercial 3D chip placement tool.

Abstract

With the rapid advancement of human science and technology, problems in industrial scenarios are becoming increasingly challenging, bringing significant challenges to traditional algorithm design. Automated algorithm design with LLMs emerges as a promising solution, but the currently adopted black-box modeling deprives LLMs of any awareness of the intrinsic mechanism of the target problem, leading to hallucinated designs. In this paper, we introduce Evolutionary Stagewise Algorithm Design (EvoStage), a novel evolutionary paradigm that bridges the gap between the rigorous demands of industrial-scale algorithm design and the LLM-based algorithm design methods. Drawing inspiration from CoT, EvoStage decomposes the algorithm design process into sequential, manageable stages and integrates real-time intermediate feedback to iteratively refine algorithm design directions. To further reduce the algorithm design space and avoid falling into local optima, we introduce a multi-agent system and a "global-local perspective" mechanism. We apply EvoStage to the design of two types of common optimizers: designing parameter configuration schedules of the Adam optimizer for chip placement, and designing acquisition functions of Bayesian optimization for black-box optimization. Experimental results across open-source benchmarks demonstrate that EvoStage outperforms human-expert designs and existing LLM-based methods within only a couple of evolution steps, even achieving the historically state-of-the-art half-perimeter wire-length results on every tested chip case. Furthermore, when deployed on a commercial-grade 3D chip placement tool, EvoStage significantly surpasses the original performance metrics, achieving record-breaking efficiency. We hope EvoStage can significantly advance automated algorithm design in the real world, helping elevate human productivity.