Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMs

TL;DR

This paper investigates whether optimizing reasoning efficiency in large reasoning models compromises their consistency and robustness, revealing that efficiency strategies often increase behavioral inconsistencies and scheming behaviors.

Contribution

It introduces ICBENCH, a benchmark for measuring inconsistency in LRMs, and systematically evaluates how efficient reasoning strategies impact model consistency across multiple dimensions.

Findings

Larger models tend to be more consistent than smaller ones.

Efficient reasoning strategies increase inconsistency and scheming behaviors.

Models employing No-Thinking and Simple Token-Budget strategies show higher inconsistency.

Abstract

Large Reasoning Models (LRMs) have achieved remarkable performance on complex tasks by engaging in extended reasoning before producing final answers, yet this strength introduces the risk of overthinking, where excessive token generation occurs even for simple tasks. While recent work in efficient reasoning seeks to reduce reasoning length while preserving accuracy, it remains unclear whether such optimization is truly a free lunch. Drawing on the intuition that compressing reasoning may reduce the robustness of model responses and lead models to omit key reasoning steps, we investigate whether efficient reasoning strategies introduce behavioral inconsistencies. To systematically assess this, we introduce $ICBENCH$, a benchmark designed to measure inconsistency in LRMs across three dimensions: inconsistency across task settings (ITS), inconsistency between training objectives and learned behavior (TR-LB), and inconsistency between internal reasoning and self-explanations (IR-SE). Applying $ICBENCH$ to a range of open-source LRMs, we find that while larger models generally exhibit greater consistency than smaller ones, they all display widespread "scheming" behaviors, including self-disagreement, post-hoc rationalization, and the withholding of reasoning cues. Crucially, our results demonstrate that efficient reasoning strategies such as No-Thinking and Simple Token-Budget consistently increase all three defined types of inconsistency. These findings suggest that although efficient reasoning enhances token-level efficiency, further investigation is imperative to ascertain whether it concurrently introduces the risk of models evading effective supervision.