Evaluating Robustness of Reasoning Models on Parameterized Logical Problems

TL;DR

This paper introduces a diagnostic benchmark for 2-SAT to evaluate the reasoning robustness of language models, isolating structural factors that influence satisfiability and revealing brittleness under targeted perturbations.

Contribution

It presents a structured, parameterized 2-SAT benchmark that isolates reasoning competencies and failure modes, enabling detailed analysis of model robustness beyond surface-level difficulty.

Findings

Models exhibit sharp performance drops under structural perturbations.

Surface statistics alone do not predict model robustness.

Structural interventions reveal brittleness regimes in reasoning models.

Abstract

Logic provides a controlled testbed for evaluating LLM-based reasoners, yet standard SAT-style benchmarks often conflate surface difficulty (length, wording, clause order) with the structural phenomena that actually determine satisfiability. We introduce a diagnostic benchmark for 2-SAT built from parameterized families of structured 2--CNF formulas, where satisfiability is characterized by the implication graph and can be tuned along interpretable axes. Our generators isolate distinct competencies and failure modes: (i) contradiction-cycle UNSAT cores with controllable size and imbalance, (ii) SAT instances with a prescribed fraction of free variables to control solution multiplicity, (iii) planted backbones that modulate propagation, (iv) late bridge clauses that couple otherwise monotone regions to probe sensitivity to ordering and revision, and (v) symmetry/duplication variants that test abstraction under renaming and redundant structure. We evaluate LLM-based reasoners on decision accuracy and assignment validity, and quantify robustness under semantics-preserving perturbations such as clause reordering, filler clauses, and variable renaming. Across models, we observe sharp performance transitions under targeted structural interventions even when surface statistics are held fixed, revealing brittleness regimes that are invisible to aggregate SAT accuracy.