Lessons from Training Grounded LLMs with Verifiable Rewards

TL;DR

This paper demonstrates that reinforcement learning with verifiable rewards and internal reasoning significantly improves the grounding, answer correctness, and citation quality of large language models, especially on unanswerable and complex queries.

Contribution

It introduces a two-stage training method using GRPO for outcome-based rewards and combines it with instruction tuning, advancing the reliability of grounded LLM responses.

Findings

Models with reasoning and RL outperform instruction-only models.

Two-stage training stabilizes learning and improves grounding.

Combining GPT-4 distillation with GRPO enhances long-form QA performance.

Abstract

Generating grounded and trustworthy responses remains a key challenge for large language models (LLMs). While retrieval-augmented generation (RAG) with citation-based grounding holds promise, instruction-tuned models frequently fail even in straightforward scenarios: missing explicitly stated answers, citing incorrectly, or refusing when evidence is available. In this work, we explore how reinforcement learning (RL) and internal reasoning can enhance grounding in LLMs. We use the GRPO (Group Relative Policy Optimization) method to train models using verifiable outcome-based rewards targeting answer correctness, citation sufficiency, and refusal quality, without requiring gold reasoning traces or expensive annotations. Through comprehensive experiments across ASQA, QAMPARI, ELI5, and ExpertQA we show that reasoning-augmented models significantly outperform instruction-only variants, especially in handling unanswerable queries and generating well-cited responses. A two-stage training setup, first optimizing answer and citation behavior and then refusal, further improves grounding by stabilizing the learning signal. Additionally, we revisit instruction tuning via GPT-4 distillation and find that combining it with GRPO enhances performance on long-form, generative QA tasks. Overall, our findings highlight the value of reasoning, stage-wise optimization, and outcome-driven RL for building more verifiable and reliable LLMs.