Towards Transparent RAG: Fostering Evidence Traceability in LLM Generation via Reinforcement Learning

TL;DR

This paper introduces TRACE, a reinforcement learning framework that enhances evidence traceability in retrieval-augmented generation by guiding large language models to produce transparent, evidence-backed responses, improving interpretability and accuracy.

Contribution

The paper presents a novel RL-based method, TRACE, for improving evidence traceability in LLMs, with new strategies for reward merging and training stability.

Findings

Achieves 10-30% accuracy improvements with transparent evidence citations

Demonstrates strong generalization to unseen tasks

Performs comparably to advanced commercial LLMs

Abstract

Retrieval-Augmented Generation (RAG) delivers substantial value in knowledge-intensive applications. However, its generated responses often lack transparent reasoning paths that trace back to source evidence from retrieved documents. This opacity not only compromises the interpretability of the output but also limits the model's ability to fully exploit the provided context. To address this, we propose TRACE (Transparent RAG with evidenCE tracing), a framework designed to enhance evidence traceability in Large Language Models (LLMs) through reinforcement learning (RL). TRACE guides LLMs to produce structured outputs with explicit evidence citations by prompting and rewarding evidence relevance and proper formatting, alongside accuracy, to optimize structured traceability. To ensure training stability with multiple reward signals, we further introduce an adaptive strategy for merging rewards and adopt a stabilized KL-divergence estimator. Experiments on three multi-hop QA datasets using Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct show that TRACE achieves both transparent, evidence-attributed outputs and accuracy improvements of 10-30%. The resulting performance is comparable to advanced commercial LLMs (e.g., OpenAI o1, DeepSeek-R1). Further analyses demonstrate strong generalization capabilities to unseen tasks. Our code is publicly available now.