Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs

TL;DR

This paper introduces Graph-RFT, a reinforcement learning framework that enhances complex knowledge graph question answering by enabling autonomous planning and adaptive retrieval from knowledge graphs and web sources, improving reasoning over incomplete knowledge.

Contribution

It proposes a novel two-stage reinforcement fine-tuning approach with structured planning and retrieval, addressing limitations of existing KGQA methods in complex scenarios.

Findings

Improved reasoning accuracy over incomplete knowledge graphs.

Effective multi-source retrieval scheduling guided by reinforcement learning.

Enhanced multi-step reasoning with structured planning modules.

Abstract

Knowledge Graph Question Answering aims to answer natural language questions by reasoning over structured knowledge graphs. While large language models have advanced KGQA through their strong reasoning capabilities, existing methods continue to struggle to fully exploit both the rich knowledge encoded in KGs and the reasoning capabilities of LLMs, particularly in complex scenarios. They often assume complete KG coverage and lack mechanisms to judge when external information is needed, and their reasoning remains locally myopic, failing to maintain coherent multi-step planning, leading to reasoning failures even when relevant knowledge exists. We propose Graph-RFT, a novel two-stage reinforcement fine-tuning KGQA framework with a 'plan-KGsearch-and-Websearch-during-think' paradigm, that enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions. Graph-RFT introduces a chain-of-thought fine-tuning method with a customized plan-retrieval dataset activates structured reasoning and resolves the GRPO cold-start problem. It then introduces a novel plan-retrieval guided reinforcement learning process integrates explicit planning and retrieval actions with a multi-reward design, enabling coverage-aware retrieval scheduling. It employs a Cartesian-inspired planning module to decompose complex questions into ordered subquestions, and logical expression to guide tool invocation for globally consistent multi-step reasoning. This reasoning retrieval process is optimized with a multi-reward combining outcome and retrieval specific signals, enabling the model to learn when and how to combine KG and web retrieval effectively.