ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction

TL;DR

This paper introduces ARCHE, a new task and benchmark for evaluating large language models' ability to explicitly extract and categorize reasoning steps into standard paradigms, revealing current models' limitations in scientific reasoning.

Contribution

The paper presents ARCHE, a novel task and benchmark for explicit reasoning chain extraction, along with logic-aware metrics and an evaluation of 10 leading LLMs.

Findings

Models show a trade-off between content coverage and logical validity.

No current model can fully extract complete, standard reasoning chains.

Significant gap exists between current LLM reasoning abilities and scientific inference requirements.

Abstract

Large language models (LLMs) are increasingly used in scientific domains. While they can produce reasoning-like content via methods such as chain-of-thought prompting, these outputs are typically unstructured and informal, obscuring whether models truly understand the fundamental reasoning paradigms that underpin scientific inference. To address this, we introduce a novel task named Latent Reasoning Chain Extraction (ARCHE), in which models must decompose complex reasoning arguments into combinations of standard reasoning paradigms in the form of a Reasoning Logic Tree (RLT). In RLT, all reasoning steps are explicitly categorized as one of three variants of Peirce's fundamental inference modes: deduction, induction, or abduction. To facilitate this task, we release ARCHE Bench, a new benchmark derived from 70 Nature Communications articles, including more than 1,900 references and 38,000 viewpoints. We propose two logic-aware evaluation metrics: Entity Coverage (EC) for content completeness and Reasoning Edge Accuracy (REA) for step-by-step logical validity. Evaluations on 10 leading LLMs on ARCHE Bench reveal that models exhibit a trade-off between REA and EC, and none are yet able to extract a complete and standard reasoning chain. These findings highlight a substantial gap between the abilities of current reasoning models and the rigor required for scientific argumentation.