Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation

TL;DR

This paper introduces Reinforced Hesitation, a training method for language models that encourages them to abstain when uncertain, improving trustworthiness by reducing hallucinations and enabling better risk management.

Contribution

It proposes Reinforced Hesitation with ternary rewards for training models to abstain, and introduces inference strategies that leverage abstention for safer, more trustworthy responses.

Findings

Models trained with RH can effectively balance accuracy and abstention.

Abstention strategies outperform majority voting in reducing errors.

Reinforced Hesitation creates models that are more honest about their limitations.

Abstract

Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -$\lambda$ error) instead of binary. Controlled experiments on logic puzzles reveal that varying $\lambda$ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know'' from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.