Enhancing Reliability across Short and Long-Form QA via Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning framework that reduces hallucinations in large language models for short and long-form QA, improving reliability without sacrificing reasoning ability.

Contribution

It presents a novel RL approach that mitigates both intrinsic and extrinsic hallucinations and encourages cautiousness in answering unanswerable questions.

Findings

Significant reduction in hallucinations across benchmarks

Improved factual accuracy and reliability

Enhanced model cautiousness in unanswerable cases

Abstract

While reinforcement learning has unlocked unprecedented complex reasoning in large language models, it has also amplified their propensity for hallucination, creating a critical trade-off between capability and reliability. This work confronts this challenge by introducing a targeted RL framework designed to mitigate both intrinsic and extrinsic hallucinations across short and long-form question answering. We address extrinsic hallucinations (flawed internal knowledge) by creating a novel training set from open-ended conversions of TriviaQA. Concurrently, we tackle intrinsic hallucinations (unfaithfulness to context) by leveraging long-form texts from FineWeb in a fact-grounding reward scheme. To further bolster reliability, our framework explicitly rewards the model for refusing to answer unanswerable questions, thereby cultivating crucial cautiousness. Extensive experiments demonstrate that our methodology yields significant performance gains across a diverse suite of benchmarks, substantially reducing both hallucination types. Ultimately, this research contributes a practical framework for resolving the critical tension between advanced reasoning and factual trustworthiness, paving the way for more capable and reliable large language models.