From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?

TL;DR

This paper introduces AR-Bench, a new benchmark to evaluate large language models' active reasoning abilities, revealing their significant challenges in acquiring and using missing information in interactive scenarios.

Contribution

The paper presents AR-Bench, a novel benchmark for assessing active reasoning in LLMs, and provides empirical evidence of their limitations in real-world, interactive tasks.

Findings

LLMs struggle with active reasoning tasks in AR-Bench.

Advanced strategies only modestly improve active reasoning performance.

There is a critical need for new training methodologies for active reasoning.

Abstract

While existing benchmarks probe the reasoning abilities of large language models (LLMs) across diverse domains, they predominantly assess passive reasoning, providing models with all the information needed to reach a solution. By contrast, active reasoning-where an LLM must interact with external systems to acquire missing evidence or data-has received little systematic attention. To address this shortfall, we present AR-Bench, a novel benchmark designed explicitly to evaluate an LLM's active reasoning skills. AR-Bench comprises three task families-detective cases, situation puzzles, and guessing numbers-that together simulate real-world, agentic scenarios and measure performance across commonsense, logical, and symbolic reasoning challenges. Empirical evaluation on AR-Bench demonstrates that contemporary LLMs exhibit pronounced difficulties with active reasoning: they frequently fail to acquire or leverage the information needed to solve tasks. This gap highlights a stark divergence between their passive and active reasoning abilities. Moreover, ablation studies indicate that even advanced strategies, such as tree-based searching or post-training approaches, yield only modest gains and fall short of the levels required for real-world deployment. Collectively, these findings highlight the critical need to advance methodology for active reasoning, e.g., incorporating interactive learning, real-time feedback loops, and environment-aware objectives for training. The benchmark is publicly available at: https://github.com/tmlr-group/AR-Bench.