I-CALM: Incentivizing Confidence-Aware Abstention for LLM Hallucination Mitigation

TL;DR

This paper introduces I-CALM, a prompt-based framework that incentivizes confidence-aware abstention in LLMs to reduce hallucinations without model retraining, by eliciting confidence, rewarding abstention, and applying normative principles.

Contribution

The paper presents a novel prompt-only approach that improves factual answer reliability by encouraging abstention based on confidence, without modifying the underlying model.

Findings

Confidence-eliciting prompts reduce false-answer rate.

Normative principles further improve abstention accuracy.

Framework trades coverage for increased answer reliability.

Abstract

Large language models (LLMs) frequently produce confident but incorrect answers, partly because common binary scoring conventions reward answering over honestly expressing uncertainty. We study whether prompt-only interventions -- explicitly announcing reward schemes for answer-versus-abstain decisions plus humility-oriented normative principles -- can reduce hallucination risk without modifying the model. Our focus is epistemic abstention on factual questions with a verifiable answer, where current LLMs often fail to abstain despite being uncertain about their answers. We first assess self-reported verbal confidence as a usable uncertainty signal, showing stability under prompt paraphrasing and reasonable calibration against a token-probability baseline. We then study I-CALM, a prompt-based framework that (i) elicits verbal confidence, (ii) partially rewards abstention through explicit reward schemes, and (iii) adds lightweight normative principles emphasizing truthfulness, humility, and responsibility. Using GPT-5 mini on PopQA as the main setting, we find that confidence-eliciting, abstention-rewarding prompts, especially with norms, reduce the false-answer rate on answered cases mainly by identifying and shifting error-prone cases to abstention and re-calibrating their confidence. This trades coverage for reliability while leaving forced-answer performance largely unchanged. Varying the abstention reward yields a clear abstention-hallucination frontier. Overall, results show the framework can improve selective answering on factual questions without retraining, with the magnitude of effect varying across models and datasets. Code is available at the following https://github.com/binzeli/hallucinationControl.