ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization

TL;DR

This paper introduces ConstraintBench, a benchmark for evaluating large language models' ability to directly solve constrained optimization problems across multiple domains, revealing current models' limitations in feasibility and optimality.

Contribution

ConstraintBench provides a comprehensive evaluation framework for LLMs on direct constrained optimization, including ground-truth solutions verified by Gurobi, and highlights key challenges in feasibility and optimality.

Findings

Best model achieves 65.0% feasibility

Feasible solutions reach 89-96% of optimal objective

Models struggle with joint feasibility and optimality

Abstract

Large language models are increasingly applied to operational decision-making where the underlying structure is constrained optimization. Existing benchmarks evaluate whether LLMs can formulate optimization problems as solver code, but leave open a complementary question. Can LLMs directly produce correct solutions to fully specified constrained optimization problems without access to a solver? We introduce ConstraintBench, a benchmark for evaluating LLMs on direct constrained optimization across 10 operations research domains, with all ground-truth solutions verified by the Gurobi solver. Each task presents a natural-language scenario with entities, constraints, and an optimization objective; the model must return a structured solution that a deterministic verifier checks against every constraint and the solver-proven optimum. We evaluate six frontier models on 200 tasks and find that feasibility, not optimality, is the primary bottleneck. The best model achieves only 65.0% feasibility, yet feasible solutions average 89 to 96% of the Gurobi-optimal objective. No model exceeds 30.5% on joint feasibility and optimality within 0.1% of the solver reference. Per-domain analysis shows large variation in difficulty, with average feasibility spanning from 85.0% in the facility location domain to 0.8% in the crew assignment domain. Further, systematic failure modes include duration constraint misunderstanding, entity hallucination, and a feasibility-optimality decoupling in facility location and vehicle routing where models achieve high feasibility but 0% optimality. ConstraintBench and all evaluation infrastructure will be publicly released.