Reinforcement Learning Enhanced Multi-hop Reasoning for Temporal Knowledge Question Answering

TL;DR

This paper introduces the MRE framework that enhances multi-hop reasoning in temporal knowledge graph question answering using prompt-guided trajectory generation, supervised fine-tuning, and a recursive exploration method, leading to improved accuracy and robustness.

Contribution

The paper proposes a novel MRE framework with Tree-Group Relative Policy Optimization for better multi-hop reasoning in TKGQA, outperforming existing methods.

Findings

Outperforms state-of-the-art on TKGQA benchmarks.

Improves interpretability and robustness to noisy data.

Enhances reasoning trajectory selection and exploration.

Abstract

Temporal knowledge graph question answering (TKGQA) involves multi-hop reasoning over temporally constrained entity relationships in the knowledge graph to answer a given question. However, at each hop, large language models (LLMs) retrieve subgraphs with numerous temporally similar and semantically complex relations, increasing the risk of suboptimal decisions and error propagation. To address these challenges, we propose the multi-hop reasoning enhanced (MRE) framework, which enhances both forward and backward reasoning to improve the identification of globally optimal reasoning trajectories. Specifically, MRE begins with prompt engineering to guide the LLM in generating diverse reasoning trajectories for a given question. Valid reasoning trajectories are then selected for supervised fine-tuning, serving as a cold-start strategy. Finally, we introduce Tree-Group Relative Policy Optimization (T-GRPO), a recursive, tree-structured learning-by-exploration approach. At each hop, exploration establishes strong causal dependencies on the previous hop, while evaluation is informed by multi-path exploration feedback from subsequent hops. Experimental results on two TKGQA benchmarks indicate that the proposed MRE-based model consistently surpasses state-of-the-art (SOTA) approaches in handling complex multi-hop queries. Further analysis highlights improved interpretability and robustness to noisy temporal annotations.