Back to the Beginning of Heuristic Design: Bridging Code and Knowledge with LLMs

TL;DR

This paper introduces a knowledge-first approach to automatic heuristic design for combinatorial optimization using large language models, emphasizing explicit, reusable knowledge over code-centric methods.

Contribution

It formalizes a top-down, knowledge-centric paradigm that enhances efficiency, transferability, and generalization in heuristic discovery, outperforming traditional code-focused approaches.

Findings

Knowledge-first search improves discovery efficiency.

Knowledge-centric methods outperform code-centric pipelines.

Combining both strategies yields further gains.

Abstract

Large language models (LLMs) have recently advanced automatic heuristic design (AHD) for combinatorial optimization (CO), where candidate heuristics are iteratively proposed, evaluated, and refined. Most existing approaches search over executable programs and distill insights from execution feedback to guide later iterations. Because this process moves from low-level implementations to high-level principles, we refer to it as a bottom-up paradigm. We argue that this view is incomplete and introduce a complementary top-down perspective: knowledge becomes the primary search object and code merely instantiates and tests it, making what is learned explicit and reusable across problems and trajectories. We formalize this shift through a statistical-learning view that exposes a distortion--compression trade-off, and instantiate it in both population-based and tree-based AHD frameworks. Across CO and tasks beyond it, knowledge-first search improves discovery efficiency, transfer, and generalization, often outperforming code-centric pipelines, while combining both strategies yields further gains. Our results suggest that progress in AHD depends on iteratively constructing and evolving interpretable hypotheses that retain value beyond a single search trajectory.