Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens

TL;DR

This paper introduces a taxonomy and an annotation framework to analyze the reasoning processes of large reasoning models from a human cognitive perspective, revealing insights for improving their training and evaluation.

Contribution

It presents a novel interdisciplinary taxonomy of reasoning steps, a large labeled dataset, and an automatic annotation tool to better understand and enhance LRM reasoning capabilities.

Findings

Post-answer self-checks are superficial and rarely lead to revisions.

Encouraging multi-step reflection improves reasoning quality.

CAPO achieves higher consistency with human annotations than baselines.

Abstract

Large reasoning models (LRMs) have garnered significant attention from researchers owing to their exceptional capability in addressing complex tasks. Motivated by the observed human-like behaviors in their reasoning processes, this paper introduces a comprehensive taxonomy to characterize atomic reasoning steps and probe the ``psyche'' of LRM intelligence. Specifically, it comprises five groups and seventeen categories derived from human mental processes, thereby grounding the understanding of LRMs in an interdisciplinary perspective. The taxonomy is then applied for an in-depth understanding of current LRMs, resulting in a distinct labeled dataset that comprises 277,534 atomic reasoning steps. Using this resource, we analyze contemporary LRMs and distill several actionable takeaways for improving training and post-training of reasoning models. Notably, our analysis reveals that prevailing post-answer ``double-checks'' (self-monitoring evaluations) are largely superficial and rarely yield substantive revisions. Thus, incentivizing comprehensive multi-step reflection, rather than simple self-monitoring, may offer a more effective path forward. To complement the taxonomy, an automatic annotation framework, named CAPO, is proposed to leverage large language models (LLMs) for generating the taxonomy-based annotations. Experimental results demonstrate that CAPO achieves higher consistency with human experts compared to baselines, facilitating a scalable and comprehensive analysis of LRMs from a human cognitive perspective. Together, the taxonomy, CAPO, and the derived insights provide a principled, scalable path toward understanding and advancing LRM reasoning.