Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models

TL;DR

This paper introduces FaithRL, a step-level reinforcement learning approach that uses explicit faithfulness rewards and contrastive signals to reduce hallucinations in small reasoning models, improving their reliability.

Contribution

It proposes FaithRL, a novel faithfulness-aware reinforcement learning method that enhances small reasoning models by explicitly rewarding faithful intermediate steps.

Findings

FaithRL significantly reduces hallucinations in reasoning steps.

Improves faithfulness and reliability of small reasoning models.

Effective across multiple models and benchmarks.

Abstract

As large language models become smaller and more efficient, small reasoning models (SRMs) are crucial for enabling chain-of-thought (CoT) reasoning in resource-constrained settings. However, they are prone to faithfulness hallucinations, especially in intermediate reasoning steps. Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained CoT evaluation, which can inadvertently reinforce unfaithful reasoning when the final answer is correct. To address these limitations, we propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model, together with an implicit truncated resampling strategy that generates contrastive signals from faithful prefixes. Experiments across multiple SRMs and Open-Book QA benchmarks demonstrate that FaithRL consistently reduces hallucinations in both the CoT and final answers, leading to more faithful and reliable reasoning. Code is available at https://github.com/Easy195/FaithRL.