FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximization

TL;DR

FaithRL is a reinforcement learning framework that enhances multi-step reasoning in language models by directly optimizing for reasoning faithfulness, thereby reducing hallucinations and improving answer accuracy.

Contribution

It introduces a novel faithfulness-maximization objective with step-level rewards, addressing over-confidence and spurious reasoning in LLMs.

Findings

Reduces hallucination rates across benchmarks

Maintains or improves answer correctness

Increases step-wise reasoning faithfulness

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has markedly improved the performance of Large Language Models (LLMs) on tasks requiring multi-step reasoning. However, most RLVR pipelines rely on sparse outcome-based rewards, providing little supervision over intermediate steps and thus encouraging over-confidence and spurious reasoning, which in turn increases hallucinations. To address this, we propose FaithRL, a general reinforcement learning framework that directly optimizes reasoning faithfulness. We formalize a faithfulness-maximization objective and theoretically show that optimizing it mitigates over-confidence. To instantiate this objective, we introduce a geometric reward design and a faithfulness-aware advantage modulation mechanism that assigns step-level credit by penalizing unsupported steps while preserving valid partial derivations. Across diverse backbones and benchmarks, FaithRL consistently reduces hallucination rates while maintaining (and often improving) answer correctness. Further analysis confirms that FaithRL increases step-wise reasoning faithfulness and generalizes robustly. Our code is available at https://github.com/aintdoin/FaithRL.