Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation

TL;DR

This paper comprehensively evaluates large language models on graph learning tasks, revealing their strengths in few-shot learning, domain transfer, and robustness, and compares them to traditional graph models.

Contribution

It provides the first extensive benchmark of LLMs on graph tasks, analyzing their performance, data concerns, and potential beyond traditional methods.

Findings

LLMs outperform traditional models in few-shot scenarios

Instruction-tuned LLMs show strong domain transferability

LLMs demonstrate high robustness and generalization

Abstract

In recent years, large language models (LLMs) have emerged as promising candidates for graph tasks. Many studies leverage natural language to describe graphs and apply LLMs for reasoning, yet most focus narrowly on performance benchmarks without fully comparing LLMs to graph learning models or exploring their broader potential. In this work, we present a comprehensive study of LLMs on graph learning tasks, evaluating both off-the-shelf and instruction-tuned models across a variety of scenarios. Beyond accuracy, we discuss data leakage concerns and computational overhead, and assess their performance under few-shot/zero-shot settings, domain transfer, structural understanding, and robustness. Our findings show that LLMs, particularly those with instruction tuning, greatly outperform traditional graph learning models in few-shot settings, exhibit strong domain transferability, and demonstrate excellent generalization and robustness. Our study highlights the broader capabilities of LLMs in graph learning and provides a foundation for future research.