Investigating Instruction Tuning Large Language Models on Graphs

TL;DR

This paper explores how instruction-tuned large language models can be applied to graph tasks, introducing a new dataset and analyzing the effectiveness of different graph representations, especially JSON, for improved understanding and generalization.

Contribution

It presents a new dataset of 79 graph tasks for instruction tuning LLMs and evaluates the impact of graph representation formats on model performance and generalization.

Findings

JSON format outperforms natural language and code formats for graph representation

Instruction-tuned LLMs can generalize across diverse graph tasks

Optimal graph representation enhances understanding of complex structures

Abstract

Inspired by the recent advancements of Large Language Models (LLMs) in NLP tasks, there's growing interest in applying LLMs to graph-related tasks. This study delves into the capabilities of instruction-following LLMs for engaging with real-world graphs, aiming to offer empirical insights into how LLMs can effectively interact with graphs and generalize across graph tasks. We begin by constructing a dataset designed for instruction tuning, which comprises a diverse collection of 79 graph-related tasks from academic and e-commerce domains, featuring 44,240 training instances and 18,960 test samples. Utilizing this benchmark, our initial investigation focuses on identifying the optimal graph representation that serves as a conduit for LLMs to understand complex graph structures. Our findings indicate that JSON format for graph representation consistently outperforms natural language and code formats across various LLMs and graph types. Furthermore, we examine the key factors that influence the generalization abilities of instruction-tuned LLMs by evaluating their performance on both in-domain and out-of-domain graph tasks.