Are Large Language Models In-Context Graph Learners?

TL;DR

This paper explores how large language models can be adapted to learn from graph-structured data by framing it as a retrieval-augmented generation process, significantly improving their performance on graph tasks without additional fine-tuning.

Contribution

The paper introduces RAG frameworks that enable LLMs to better handle graph data, bridging the gap between language models and graph neural networks in in-context learning.

Findings

RAG frameworks improve LLM performance on graph tasks

Significant gains in zero-shot graph learning scenarios

Effective without fine-tuning or model modification

Abstract

Large language models (LLMs) have demonstrated remarkable in-context reasoning capabilities across a wide range of tasks, particularly with unstructured inputs such as language or images. However, LLMs struggle to handle structured data, such as graphs, due to their lack of understanding of non-Euclidean structures. As a result, without additional fine-tuning, their performance significantly lags behind that of graph neural networks (GNNs) in graph learning tasks. In this paper, we show that learning on graph data can be conceptualized as a retrieval-augmented generation (RAG) process, where specific instances (e.g., nodes or edges) act as queries, and the graph itself serves as the retrieved context. Building on this insight, we propose a series of RAG frameworks to enhance the in-context learning capabilities of LLMs for graph learning tasks. Comprehensive evaluations demonstrate that our proposed RAG frameworks significantly improve LLM performance on graph-based tasks, particularly in scenarios where a pretrained LLM must be used without modification or accessed via an API.