CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs

TL;DR

CoTJudger is a graph-based framework that measures the efficiency of reasoning in large models by identifying essential reasoning steps and detecting redundancy, thus improving evaluation and diagnosis of model reasoning.

Contribution

Introduces CoTJudger, a novel graph-driven method that quantifies reasoning efficiency and redundancy in large reasoning models, enabling more precise evaluation.

Findings

Redundancy is common in large reasoning models.

CoTJudger identifies key failure modes like verification obsession.

The framework provides a practical metric to distinguish reasoning ability from computational waste.

Abstract

Large Reasoning Models (LRMs) have demonstrated strong performance by producing extended Chain-of-Thought (CoT) traces before answering. However, this paradigm often induces over-reasoning: redundant calculations and circular self-verification that increase computational cost without improving outcomes. Existing evaluations largely emphasize final accuracy or coarse token counts, and lack automated tools to separate essential logic from structural redundancy. We introduce CoTJudger, a graph-driven framework that quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path (SEP) needed to reach a correct solution. This yields an interpretable efficiency signal -- how much of a CoT is necessary versus structurally redundant -- that is comparable across models and tasks. Evaluating 21 LRMs, CoTJudger reveals pervasive redundancy and surfaces recurring failure modes, including verification obsession and compensatory redundancy. These results provide a practical metric for disentangling reasoning ability from computational waste, enabling more targeted evaluation and diagnosis of LRM efficiency.