DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning

TL;DR

DynaSearcher is a novel search agent that uses dynamic knowledge graphs and multi-reward reinforcement learning to improve factual consistency, efficiency, and answer quality in multi-step retrieval tasks, achieving state-of-the-art results with limited resources.

Contribution

The paper introduces DynaSearcher, integrating dynamic knowledge graphs with multi-reward RL to enhance search accuracy and efficiency in LLM-based retrieval systems, addressing key practical challenges.

Findings

Achieves state-of-the-art accuracy on six multi-hop QA datasets.

Maintains high performance with small-scale models and limited resources.

Demonstrates strong generalization and robustness across environments.

Abstract

Multi-step agentic retrieval systems based on large language models (LLMs) have demonstrated remarkable performance in complex information search tasks. However, these systems still face significant challenges in practical applications, particularly in generating factually inconsistent intermediate queries and inefficient search trajectories, which can lead to reasoning deviations or redundant computations. To address these issues, we propose DynaSearcher, an innovative search agent enhanced by dynamic knowledge graphs and multi-reward reinforcement learning (RL). Specifically, our system leverages knowledge graphs as external structured knowledge to guide the search process by explicitly modeling entity relationships, thereby ensuring factual consistency in intermediate queries and mitigating biases from irrelevant information. Furthermore, we employ a multi-reward RL framework for fine-grained control over training objectives such as retrieval accuracy, efficiency, and response quality. This framework promotes the generation of high-quality intermediate queries and comprehensive final answers, while discouraging unnecessary exploration and minimizing information omissions or redundancy. Experimental results demonstrate that our approach achieves state-of-the-art answer accuracy on six multi-hop question answering datasets, matching frontier LLMs while using only small-scale models and limited computational resources. Furthermore, our approach demonstrates strong generalization and robustness across diverse retrieval environments and larger-scale models, highlighting its broad applicability.