UR$^2$: Unify RAG and Reasoning through Reinforcement Learning

TL;DR

UR$^2$ is a reinforcement learning framework that unifies retrieval-augmented generation and reasoning, enhancing large language models' performance across diverse tasks by dynamically coordinating retrieval and reasoning strategies.

Contribution

The paper introduces UR$^2$, a novel framework that combines retrieval and reasoning with a difficulty-aware curriculum and hybrid knowledge access, improving robustness and generalization.

Findings

UR$^2$ outperforms existing RAG and RL baselines on multiple tasks.

UR$^2$ achieves performance comparable to GPT-4 on several benchmarks.

The code for UR$^2$ is publicly available at the provided GitHub link.

Abstract

Large Language Models (LLMs) have shown strong capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG) for knowledge grounding and Reinforcement Learning from Verifiable Rewards (RLVR) for complex reasoning. However, existing attempts to unify these paradigms remain narrow in scope, typically limited to open-domain QA with fixed retrieval settings, which constrains generalization to broader domains. To address this limitation, we propose UR$^2$ (Unified RAG and Reasoning)), a general reinforcement learning framework that dynamically coordinates retrieval and reasoning. UR$^2$ introduces two key designs: a difficulty-aware curriculum that selectively invokes retrieval only for challenging instances, and a hybrid knowledge access strategy that combines domain-specific offline corpora with on-the-fly LLM-generated summaries. Together, these components mitigate the imbalance between retrieval and reasoning and improve robustness to noisy information. Experiments on open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks show that UR$^2$, built on Qwen-2.5-3/7B and LLaMA-3.1-8B, consistently outperforms existing RAG and RL baselines, and achieves performance comparable to GPT-4o-mini and GPT-4.1-mini on several benchmarks. Our code is available at https://github.com/Tsinghua-dhy/UR2.