Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization

TL;DR

This paper introduces NLCO, a comprehensive benchmark for evaluating large language models on natural language combinatorial optimization tasks, revealing their strengths and limitations across various problem types and sizes.

Contribution

The paper presents NLCO, a new benchmark with 43 problems and a detailed taxonomy, enabling fine-grained evaluation of LLMs on combinatorial optimization in natural language.

Findings

High-performing models solve small instances well but struggle as size increases.

Set-based tasks are easier for LLMs than graph-structured problems.

Model performance degrades with larger instances, even with more reasoning tokens.

Abstract

While large language models (LLMs) have shown strong performance in math and logic reasoning, their ability to handle combinatorial optimization (CO) -- searching high-dimensional solution spaces under hard constraints -- remains underexplored. To bridge the gap, we introduce NLCO, a \textbf{N}atural \textbf{L}anguage \textbf{C}ombinatorial \textbf{O}ptimization benchmark that evaluates LLMs on end-to-end CO reasoning: given a language-described decision-making scenario, the model must output a discrete solution without writing code or calling external solvers. NLCO covers 43 CO problems and is organized using a four-layer taxonomy of variable types, constraint families, global patterns, and objective classes, enabling fine-grained evaluation. We provide solver-annotated solutions and comprehensively evaluate LLMs by feasibility, solution optimality, and reasoning efficiency. Experiments across a wide range of modern LLMs show that high-performing models achieve strong feasibility and solution quality on small instances, but both degrade as instance size grows, even if more tokens are used for reasoning. We also observe systematic effects across the taxonomy: set-based tasks are relatively easy, whereas graph-structured problems and bottleneck objectives lead to more frequent failures.