Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining

TL;DR

MuseGraph is a novel framework that combines GNNs and LLMs to enable a single model to perform diverse graph mining tasks across multiple datasets, improving accuracy and generative capabilities.

Contribution

The paper introduces MuseGraph, a unified foundation model integrating GNNs and LLMs with a new instruction tuning strategy for versatile graph mining.

Findings

Significant improvements in five graph tasks across ten datasets.

Enhanced accuracy in downstream graph tasks.

Improved generative abilities of LLMs for graph data.

Abstract

Graphs with abundant attributes are essential in modeling interconnected entities and enhancing predictions across various real-world applications. Traditional Graph Neural Networks (GNNs) often require re-training for different graph tasks and datasets. Although the emergence of Large Language Models (LLMs) has introduced new paradigms in natural language processing, their potential for generic graph mining, training a single model to simultaneously handle diverse tasks and datasets, remains under-explored. To this end, our novel framework MuseGraph, seamlessly integrates the strengths of GNNs and LLMs into one foundation model for graph mining across tasks and datasets. This framework first features a compact graph description to encapsulate key graph information within language token limitations. Then, we propose a diverse instruction generation mechanism with Chain-of-Thought (CoT)-based instruction packages to distill the reasoning capabilities from advanced LLMs like GPT-4. Finally, we design a graph-aware instruction tuning strategy to facilitate mutual enhancement across multiple tasks and datasets while preventing catastrophic forgetting of LLMs' generative abilities. Our experimental results demonstrate significant improvements in five graph tasks and ten datasets, showcasing the potential of our MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while improving the generation abilities of LLMs.