GraphDancer: Training LLMs to Explore and Reason over Graphs via Curriculum Reinforcement Learning

TL;DR

GraphDancer introduces a reinforcement learning framework that enables medium-sized language models to effectively navigate and reason over complex, heterogeneous graph-structured knowledge sources through curriculum learning.

Contribution

It presents a novel RL-based training method with a graph-aware curriculum to improve LLMs' ability to explore and reason over graph-structured data.

Findings

Outperforms larger models and GPT-4o-mini on cross-domain tasks

Demonstrates robust generalization to unseen domains and question types

Effective for moderate-sized LLMs with 3B parameters

Abstract

Large language models (LLMs) increasingly rely on external knowledge to improve factuality, yet many real-world knowledge sources are organized as heterogeneous graphs rather than plain text. Reasoning over such graph-structured knowledge poses two key challenges: (1) navigating structured, schema-defined relations requires precise function calls rather than similarity-based retrieval, and (2) answering complex questions often demands multi-hop evidence aggregation through iterative information seeking. We propose GraphDancer, a reinforcement learning (RL) framework that teaches LLMs to navigate graphs by interleaving reasoning and function execution. To make RL effective for moderate-sized LLMs, we introduce a graph-aware curriculum that schedules training by the structural complexity of information-seeking trajectories using an easy-to-hard biased sampler. We evaluate GraphDancer on a multi-domain benchmark by training on one domain only and testing on unseen domains and out-of-distribution question types. Despite using only a 3B backbone, GraphDancer outperforms baselines equipped with either a 14B backbone or GPT-4o-mini, demonstrating robust cross-domain generalization of graph exploration and reasoning skills. Our code and models can be found at https://yuyangbai.com/graphdancer/ .