Learning the Boundary of Solvability: Aligning LLMs to Detect Unsolvable Problems

TL;DR

This paper introduces UnsolvableQA and UnsolvableRL to improve LLMs' ability to distinguish between unsolvable problems and capability limitations, reducing hallucinations and enhancing reliability.

Contribution

The paper presents a new dataset and reinforcement learning framework for better detection of unsolvable problems in LLMs, addressing a key reliability challenge.

Findings

Achieves over 85% unsolvability detection rate

Increases reasoning accuracy from 43.4% to 69.4%

Identifies data-training interaction effects on model robustness

Abstract

Ensuring large language model (LLM) reliability requires distinguishing objective unsolvability (inherent contradictions) from subjective capability limitations (tasks exceeding model competence). Current LLMs often conflate these dimensions, leading to hallucinations in which they return confident answers to inherently unsolvable queries. To address this issue, we propose a multi-domain dataset containing both solvable and unsolvable questions, UnsolvableQA, together with an alignment framework, UnsolvableRL. First, we construct UnsolvableQA by "Reverse Construction" that systematically injects logical contradictions into otherwise valid reasoning chains. Second, we introduce UnsolvableRL, a reinforcement learning paradigm that balances objective unsolvability detection with calibrated confidence under capability limits. Empirically, our approach achieves robust unsolvability detection (>85% detection rate) and boosts solvable reasoning accuracy from 43.4% to 69.4% on Qwen3-4B-Instruct. Crucially, we identify a data-training interaction: strict alignment constraints induce Capability Collapse without unsolvable data, but act as a regularizer for rigor when such data are included, thereby improving overall robustness. Our code and data are available at https://github.com/sfasfaffa/unsolvableQA .