GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learning

TL;DR

GlobalRAG is a reinforcement learning framework that improves multi-hop question answering by enhancing global reasoning, structured planning, and evidence utilization, leading to significant performance gains with less training data.

Contribution

It introduces a novel RL-based approach with planning and subgoal mechanisms, and new reward strategies to improve multi-hop QA reasoning and evidence use.

Findings

Achieves 14.2% higher EM and F1 scores over baselines.

Uses only 8k training data, half of what strong baselines require.

Outperforms in both in-domain and out-of-domain benchmarks.

Abstract

Reinforcement learning has recently shown promise in improving retrieval-augmented generation (RAG). Despite these advances, its effectiveness in multi-hop question answering (QA) remains limited by two fundamental limitations: (i) global planning absence to structure multi-step reasoning, and (ii) unfaithful execution, which hinders effective query formulation and consistent use of retrieved evidence. We propose GlobalRAG, a reinforcement learning framework designed to enhance global reasoning in multi-hop QA. GlobalRAG decomposes questions into subgoals, coordinates retrieval with reasoning, and refines evidence iteratively. To guide this process, we introduce Planning Quality Reward and SubGoal Completion Reward, which encourage coherent planning and reliable subgoal execution. In addition, a progressive weight annealing strategy balances process-oriented and outcome-based objectives. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that GlobalRAG significantly outperforms strong baselines while using only 8k training data (42% of the training data used by strong baselines), achieving average improvements of 14.2% in both EM and F1.