Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding

TL;DR

This paper systematically evaluates whether Graph-Tokenizing Large Language Models truly understand graph tokens, revealing their limitations and the need for further improvements in instruction tuning.

Contribution

It introduces GTEval, a unified evaluation framework, and provides extensive analysis showing current GTokenLLMs' incomplete understanding of graph tokens.

Findings

GTokenLLMs do not fully understand graph tokens.

They show over-sensitivity or over-insensitivity to instruction changes.

Additional instruction tuning improves performance but does not fully solve understanding issues.

Abstract

The remarkable success of large language models (LLMs) has motivated researchers to adapt them as universal predictors for various graph tasks. As a widely recognized paradigm, Graph-Tokenizing LLMs (GTokenLLMs) compress complex graph data into graph tokens and treat them as prefix tokens for querying LLMs, leading many to believe that LLMs can understand graphs more effectively and efficiently. In this paper, we challenge this belief: \textit{Do GTokenLLMs fully understand graph tokens in the natural-language embedding space?} Motivated by this question, we formalize a unified framework for GTokenLLMs and propose an evaluation pipeline, \textbf{GTEval}, to assess graph-token understanding via instruction transformations at the format and content levels. We conduct extensive experiments on 6 representative GTokenLLMs with GTEval. The primary findings are as follows: (1) Existing GTokenLLMs do not fully understand graph tokens. They exhibit over-sensitivity or over-insensitivity to instruction changes, and rely heavily on text for reasoning; (2) Although graph tokens preserve task-relevant graph information and receive attention across LLM layers, their utilization varies across models and instruction variants; (3) Additional instruction tuning can improve performance on the original and seen instructions, but it does not fully address the challenge of graph-token understanding, calling for further improvement.