Beyond Text: A Deep Dive into Large Language Models' Ability on Understanding Graph Data

TL;DR

This paper benchmarks large language models on graph prediction tasks to evaluate their ability to understand and leverage graph structures, comparing their performance with specialized graph neural networks.

Contribution

It provides a comprehensive analysis of LLMs' capabilities on graph data, highlighting their strengths and limitations relative to specialized graph models.

Findings

LLMs show potential in processing graph data but underperform compared to specialized models.

Prompt formatting significantly affects LLM performance on graph tasks.

Insights suggest avenues for enhancing LLMs for graph analytics.

Abstract

Large language models (LLMs) have achieved impressive performance on many natural language processing tasks. However, their capabilities on graph-structured data remain relatively unexplored. In this paper, we conduct a series of experiments benchmarking leading LLMs on diverse graph prediction tasks spanning node, edge, and graph levels. We aim to assess whether LLMs can effectively process graph data and leverage topological structures to enhance performance, compared to specialized graph neural networks. Through varied prompt formatting and task/dataset selection, we analyze how well LLMs can interpret and utilize graph structures. By comparing LLMs' performance with specialized graph models, we offer insights into the strengths and limitations of employing LLMs for graph analytics. Our findings provide insights into LLMs' capabilities and suggest avenues for further exploration in applying them to graph analytics.