Formalize, Don't Optimize: The Heuristic Trap in LLM-Generated Combinatorial Solvers

TL;DR

This paper evaluates how different representations and heuristic optimization strategies in LLM-generated combinatorial solvers affect correctness and efficiency, highlighting the risks of heuristic traps and advocating for formalization over optimization.

Contribution

It introduces the CP-SynC-XL benchmark and systematically compares solver paradigms, revealing the pitfalls of heuristic search optimization in LLM-based solver synthesis.

Findings

Python + OR-Tools achieves highest correctness across LLMs.

Heuristic search optimization yields minimal speed-ups and can reduce correctness.

Heuristic traps often cause regressions, leading to over-constraining or unverified bounds.

Abstract

Large Language Models (LLMs) struggle to solve complex combinatorial problems through direct reasoning, so recent neuro-symbolic systems increasingly use them to synthesize executable solvers. A central design question is how the LLM should represent the solver, and whether it should also attempt to optimize search. We introduce CP-SynC-XL, a benchmark of 100 combinatorial problems (4,577 instances), and evaluate three solver-construction paradigms: native algorithmic search (Python), constraint modeling through a Python solver API (Python + OR-Tools), and declarative constraint modeling (MiniZinc + OR-Tools). We find a consistent representational divergence: Python + OR-Tools attains the highest correctness across LLMs, while MiniZinc + OR-Tools has lower absolute coverage despite using the same OR-Tools back-end. Native Python is the most likely to return a schema-valid solution that fails verification, whereas solver-backed paths preserve higher conditional fidelity. On the heuristic axis, prompting for search optimization yields only small median speed-ups (1.03-1.12x) and a strongly bimodal effect: many instances slow down, and correctness drops sharply on a long tail of problems. A paired code-level audit traces these regressions to a recurring heuristic trap. Under an efficiency-oriented prompt, the LLM may replace complete search with local approximations (Python), inject unverified bounds (Python + OR-Tools), or add redundant declarative machinery that overwhelms or over-constrains the model (MiniZinc + OR-Tools). These findings support a conservative design principle for LLM-generated combinatorial solvers: use the LLM primarily to formalize variables, constraints, and objectives for verified solvers, and separately check any LLM-authored search optimization before use.