Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

TL;DR

This paper identifies that reinforcement learning fine-tuning of large language models increases hallucinations and proposes FSPO, a factuality-aware method that reduces hallucinations and improves reasoning accuracy.

Contribution

The paper introduces FSPO, a novel RL fine-tuning algorithm that incorporates factuality verification at each reasoning step to mitigate hallucinations in large language models.

Findings

FSPO reduces hallucinations effectively.

FSPO improves reasoning accuracy.

Models fine-tuned with FSPO outperform baselines.

Abstract

Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.