Awakening LLMs' Reasoning Potential: A Fine-Grained Pipeline to Evaluate and Mitigate Vague Perception

TL;DR

This paper introduces the WakenLLM pipeline to evaluate and enhance LLMs' reasoning by addressing Vague Perception, significantly improving their ability to correctly solve questions they previously abstained from, without additional training.

Contribution

The paper formalizes Vague Perception in LLMs and proposes a pipeline to measure and activate their latent reasoning potential through stage-wise metrics and stimulation techniques.

Findings

LLMs can improve up to 68.53% accuracy on Vague Perception samples using WakenLLM.

Existing reasoning baselines activate only a small portion of LLMs' potential.

Vague Perception varies across model families and sizes, informing model selection strategies.

Abstract

Large language models (LLMs) are increasingly trained to abstain on difficult questions by answering unknown. However, we observe that LLMs often misuse this option: they output unknown even when LLMs can actually solve the questions, or they fail to understand why questions are truly unsolvable. We formalize this mismatch between potential ability and the inclination of abstention as the Vague Perception phenomenon. We introduce the WakenLLM pipeline that (1) extracts Vague Perception samples and (2) measures how many of them can be converted to correct answers under stimulation. Based on stage-wise metrics (TCR, OCR, etc.) and the upper-bound accuracy Acc(WakenLLM), we quantify LLMs' reasoning potential beyond one-shot accuracy. Experiments on six LLMs suggest that, without further training or parameter revisions, LLMs can achieve up to a 68.53% increase in accuracy on Vague Perception samples through our designed pipeline. We further analyze how Vague Perception, Conformity and Degradation vary from model families and parameter sizes, and offer model selection strategies in multi-stage reasoning workflows. Finally, by comparing WakenLLM against mainstream reasoning baselines, both training and non-training ones, we show that existing baselines only activate a small portion of LLMs' reasoning potential, pointing to perception-aware reasoning as a promising direction for future LLM designing. Code and datasets are available at https://github.com/WakenLLMTeam/WakenLLM-toolkit.