Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks

TL;DR

This paper introduces EstGraph, a benchmark dataset and tasks for evaluating LLMs on large-scale graph property estimation using random walk sampling within context length constraints.

Contribution

It proposes a new benchmark and prompt strategies for assessing LLM reasoning on large, accessible graphs, addressing limitations of existing small-graph benchmarks.

Findings

LLMs can estimate properties of large graphs using random walk-based prompts.

The proposed prompts enable LLMs to handle graphs with millions of nodes.

The benchmark reveals strengths and limitations of LLM reasoning on large graph data.

Abstract

With the rapidly improving reasoning abilities of Large Language Models (LLMs), there is also a rising demand to use them in a wide variety of domains. This brings about the need to carefully evaluate the limits of the capabilities of these models with various tests and benchmarks. Graph structures are ubiquitous in real-world data, and are often used to represent and analyze relationship patterns within data. Many benchmarks have already been proposed in the graph literature to test the reasoning ability of LLMs to follow and execute graph algorithms. However, due to the limited context length of LLMs, these benchmarks consist of very small graphs. In real-world data, the size of graphs can be significantly larger, and in many cases, not fully accessible. In this paper, we examine a class of problems that arises with very large graphs having limited accessibility. We propose a large graph benchmark dataset, EstGraph, and introduce four distinct tasks designed to estimate large graph properties. We evaluate the reasoning abilities of LLMs on these tasks using a wide variety of graph datasets. In addition, we provide task-specific prompt constructions based on random walk sampling of large graphs (up to millions of nodes) that effectively convey sufficient information to LLMs within the limits of context length.