Do I Really Know? Learning Factual Self-Verification for Hallucination Reduction

TL;DR

VeriFY is a training framework that enhances large language models with self-verification capabilities to significantly reduce factual hallucinations while maintaining overall answer recall.

Contribution

It introduces a novel training-time self-verification method with stage-level loss masking to mitigate hallucinations in LLMs.

Findings

Reduces hallucination rates by up to 53.3%

Maintains high answer recall with minimal reduction

Generalizes across datasets and model scales

Abstract

Factual hallucination remains a central challenge for large language models (LLMs). Existing mitigation approaches primarily rely on either external post-hoc verification or mapping uncertainty directly to abstention during fine-tuning, often resulting in overly conservative behavior. We propose VeriFY, a training-time framework that teaches LLMs to reason about factual uncertainty through consistency-based self-verification. VeriFY augments training with structured verification traces that guide the model to produce an initial answer, generate and answer a probing verification query, issue a consistency judgment, and then decide whether to answer or abstain. To address the risk of reinforcing hallucinated content when training on augmented traces, we introduce a stage-level loss masking approach that excludes hallucinated answer stages from the training objective while preserving supervision over verification behavior. Across multiple model families and scales, VeriFY reduces factual hallucination rates by 9.7 to 53.3 percent, with only modest reductions in recall (0.4 to 5.7 percent), and generalizes across datasets when trained on a single source. The source code, training data, and trained model checkpoints will be released upon acceptance.