Adaptive Tokenization: On the Hop-Overpriority Problem in Tokenized Graph Learning Models

TL;DR

This paper introduces LGTL, a learnable token list for Graph Transformers that adaptively balances local and global information, addressing the hop-overpriority problem and improving performance on diverse graph types.

Contribution

The paper proposes LGTL, a novel plug-and-play module that replaces hand-designed token lists with an adaptive, learnable approach for better graph representation in TGLMs.

Findings

LGTL effectively addresses the hop-overpriority problem.

LGTL improves performance on heterophilic and homophilic graphs.

Experimental results validate LGTL's superiority across benchmarks.

Abstract

Graph Transformers, leveraging the global attention to capture long-range dependencies in graph structures, have significantly advanced graph machine learning, but face prohibitive computational complexity. Tokenized Graph Learning Models (TGLMs) address this issue by converting graphs into ordered token lists for scalable processing. Besides, TGLMs also empower Large Language Models (LLMs) to handle text-attributed graphs more effectively and thus are also employed in Graph LLMs. However, existing TGLMs rely on hand-designed token lists and their adaptability to diverse graph learning scenarios remains unexplored. In this paper, we first conduct extensive empirical and theoretical preliminary studies for hand-designed token lists. Surprisingly, we identify an unexplored hop-overpriority problem: the common pre-defined token lists overemphasize nearby nodes and overwhelm the ability of TGLMs to balance local and global signals. This phenomenon is especially harmful for heterophilic graphs. To address this problem, we propose the Learnable Graph Token List (LGTL), a plug-and-play module to replace hand-designed token lists in TGLMs. Specifically, LGTL adaptively adjusts the weights across hops and prioritizes informative nodes within hops through a graph attention gate module and a selection module, respectively. In this way, contextually informative nodes can be adaptively emphasized for both homophilic and heterophilic graphs. Besides, we theoretically show that LGTL can address the hop-overpriority problem. Extensive experiments on benchmarks validate the efficacy of LGTL across both Graph Transformers and Graph LLM backbones.