Budget-Efficient Automatic Algorithm Design via Code Graph

TL;DR

This paper introduces a budget-efficient method for automatic algorithm design using a graph-based approach that leverages LLMs to generate and refine algorithms through corrections, improving search efficiency.

Contribution

It formalizes a graph representation of algorithms and a correction-based search framework that enhances efficiency over traditional full-algorithm queries.

Findings

Graph-based search outperforms full-algorithm search at equal token budgets.

Corrections enable more efficient exploration of algorithm space.

Rich context helps mainly when LLM's prior knowledge is shallow.

Abstract

Large language models (LLMs) have emerged as powerful tools for automatic algorithm design (AAD). However, existing pipelines remain inefficient. They operate at the granularity of full algorithms, redundantly rewriting recurring substructures and discarding low-fitness candidates that may contain valuable algorithmic features. We formalize budget-efficient automatic algorithm design, wherein the search policy maximizes realized fitness subject to limited computational cost. We propose a directed acyclic graph representation of algorithms and build a search framework that fully exploits the LLM's output. Instead of querying the LLM for full algorithms, we use it to obtain corrections: compact operators that add, replace, or remove code blocks. Each correction augments the graph, yielding new algorithms that compose with prior corrections. This graph structure decomposes algorithms into sets of corrections, enabling correction-level credit assignment that informs subsequent queries. We complement this framework with theoretical insights into the ideal balance between search depth and breadth at different budget levels. We validate our method empirically on three combinatorial optimization problems, demonstrating consistent superiority of our graph-based search over full-algorithm search at equal token budget. Finally, our experiments suggest that rich contexts help only when the LLM's prior knowledge is shallow, and can hinder performance otherwise.