R-ConstraintBench: Evaluating LLMs on NP-Complete Scheduling

TL;DR

This paper introduces R-ConstraintBench, a scalable framework for evaluating large language models on complex, NP-Complete scheduling problems with increasing constraints, revealing their limitations in high-constraint scenarios.

Contribution

We present R-ConstraintBench, a novel benchmark for assessing LLMs on resource-constrained scheduling problems with increasing complexity and constraint interactions.

Findings

Strong models perform well on simple precedence-only tasks.

Feasibility drops significantly with additional constraint types.

Constraint interactions, not graph depth, limit model performance.

Abstract

Effective scheduling under tight resource, timing, and operational constraints underpins large-scale planning across sectors such as capital projects, manufacturing, logistics, and IT fleet transitions. However, the reliability of large language models (LLMs) when reasoning under high-constraint regimes is insufficiently characterized. To address this gap, we present R-ConstraintBench, a scalable framework that evaluates models on Resource-Constrained Project Scheduling Problems (RCPSP), an NP-Complete feasibility class, while difficulty increases via linear growth in constraints. R-ConstraintBench incrementally increases non-redundant precedence constraints in Directed Acyclic Graphs (DAGs) and then introduces downtime, temporal windows, and disjunctive constraints. As an illustrative example, we instantiate the benchmark in a data center migration setting and evaluate multiple LLMs using feasibility and error analysis, identifying degradation thresholds and constraint types most associated with failure. Empirically, strong models are near-ceiling on precedence-only DAGs, but feasibility performance collapses when downtime, temporal windows, and disjunctive constraints interact, implicating constraint interaction, not graph depth, as the principal bottleneck. Performance on clean synthetic ramps also does not guarantee transfer to domain-grounded scenarios, underscoring limited generalization.