MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs

TL;DR

MathConstraint is a new adaptive benchmark for testing the combinatorial reasoning of large language models, combining constraint satisfaction problems with solver-based verification to generate challenging, verifiable instances.

Contribution

It introduces a scalable, parameterized benchmark generator that creates difficult, automatically verifiable combinatorial reasoning problems for LLM evaluation.

Findings

Frontier models achieve 72.6% to 87.6% accuracy on easy instances.

Model accuracy drops to 18.5% to 66.9% on the main benchmark.

Tool access significantly improves model performance, with up to 52 percentage points gain.

Abstract

We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfaction problems with rigorous solver-based verification and design an adaptive generator to create instances that remain challenging as the LLMs improve in their reasoning capabilities. Unlike existing benchmarks that quickly saturate on fixed datasets or use LLM-as-a-judge for checking solutions,MathConstraint uses parameterized problem types that enable scalable generation of arbitrarily difficult and automatically verifiable instances. We release MathConstraint-Easy ($266$ instances), on which frontier models achieve between $72.6\%$ (gemini-3.1-flash-lite) and $87.6\%$ (gpt-5.5) accuracy, and MathConstraint ($329$ instances) on which the same models drop to between $18.5\%$ (claude-4.6-sonnet) and $66.9\%$ (gpt-5.5) accuracy, demonstrating the resilience of our benchmark generator against rapid progress in LLM reasoning capabilities. We evaluate 12 frontier and open-weight models with and without access to a sandboxed Python environment that includes generic SAT/SMT solvers. Tool access roughly doubles frontier accuracy on MathConstraint (mean $+28$pp; up to $+52$pp for claude-4.6-sonnet). Further, halving the tool-call budget from $8$ to $4$ rounds erases up to $37$ points -- a sensitivity that most single-budget benchmarks miss. We release the generator, dataset, and evaluation harness as a robust environment for studying combinatorial reasoning and tool-use behavior under adversarially-tunable difficulty.