ReEfBench: Quantifying the Reasoning Efficiency of LLMs

TL;DR

This paper introduces a neuro-symbolic framework to evaluate LLM reasoning, distinguishing genuine reasoning from verbosity, and analyzes factors affecting reasoning performance and limitations.

Contribution

It presents a novel process-centric evaluation method for LLM reasoning, identifies behavioral prototypes, and diagnoses failure modes and constraints.

Findings

Extended token generation is not necessary for deep reasoning

Mixing long and short CoT data can cause saturation and collapse

Distillation captures behavioral length but not logical efficacy

Abstract

Test-time scaling has enabled Large Language Models (LLMs) to tackle complex reasoning, yet the limitations of current Chain-of-Thought (CoT) evaluation obscures whether performance gains stem from genuine reasoning or mere verbosity. To address this, (1) we propose a novel neuro-symbolic framework for the non-intrusive, comprehensive process-centric evaluation of reasoning. (2) Through this lens, we identify four distinct behavioral prototypes and diagnose the failure modes. (3) We examine the impact of inference mode, training strategy, and model scale. Our analysis reveals that extended token generation is not a prerequisite for deep reasoning. Furthermore, we reveal critical constraints: mixing long and short CoT data in training risks in premature saturation and collapse, while distillation into smaller models captures behavioral length but fails to replicate logical efficacy due to intrinsic capacity limits.