GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capability

TL;DR

This paper introduces GraphInstruct, a comprehensive benchmark for graph reasoning tasks, and develops GraphSolver and GraphSolver+ models that significantly improve large language models' understanding and reasoning capabilities on graph data.

Contribution

The paper presents a new benchmark for graph reasoning, along with instruction-tuned models that enhance LLMs' graph understanding and multi-step reasoning abilities.

Findings

GraphSolver outperforms other open-source LLMs in graph understanding.

GraphSolver+ with label-mask training further improves multi-step reasoning.

Extensive experiments demonstrate the superiority of the proposed models.

Abstract

Improving the general capabilities of large language models (LLMs) is an active research topic. As a common data structure in many real-world domains, understanding graph data is a crucial part of advancing general intelligence. To this end, we propose a dynamic benchmark named GraphInstruct in this paper, which comprehensively includes 21 classical graph reasoning tasks, providing diverse graph generation pipelines and detailed intermediate reasoning steps for each sample. Based on GraphInstruct, we develop GraphSolver via efficient instruction-tuning, which demonstrates prominent graph understanding capability compared to other open-sourced LLMs. To further endow LLMs with multi-step graph reasoning capability, we propose a label-mask training strategy and build GraphSolver+, which leverages masked supervision on intermediate reasoning tokens to emphasize crucial node-identification signals. As one of the pioneering efforts to enhance the graph understanding and reasoning abilities of LLMs, extensive experiments have demonstrated the superiority of GraphSolver and GraphSolver+ over other LLMs. We sincerely hope GraphInstruct will facilitate further research on applying LLMs to graph-structured data. Our code and data are released publicly at: https://github.com/CGCL-codes/GraphInstruct.