Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers

TL;DR

This paper investigates how different graph tokenizations affect transformer expressivity, revealing fundamental trade-offs and limitations in recovering structural information across tokenization types.

Contribution

It provides theoretical analysis of spectral, random-walk, and adjacency tokenizations, establishing depth regimes and impossibility results for transforming between them.

Findings

Random-walk tokenization is lossy for any walk length.

Spectral tokenization is lossless but ill-conditioned for local tasks.

Combining tokenizations improves structural signal extraction.

Abstract

Transformers have become a central architecture for graph learning, but their application to graphs requires first choosing a tokenization: a graph-to-token map that determines which structural information is exposed at the input. In this work, we show that this choice is a fundamental component of transformer expressivity. We examine three tokenizations that serve as building blocks for many existing graph tokenizations: spectral, random-walk, and adjacency tokenizations. We prove that different tokenizations induce distinct depth regimes: the same graph computation may be realizable by a shallow transformer under one tokenization, while requiring substantially larger depth under another. For example, we prove that random-walk tokenization is lossy for any walk length, making it impossible in general to recover the graph from it, and that while spectral tokenization is lossless, it is ill-conditioned for local tasks. We further show that although both random-walk and spectral tokenizations are derived from adjacency information, it is impossible for a limited-depth transformer to convert between tokenization families in general. In particular, we establish lower bounds and impossibility results showing that unfavorable tokenizations may preclude the efficient recovery of more suitable structural representations. Finally, we complement our theory with controlled experiments on synthetic and real-world tasks, validating the predicted separations and showing that different tasks favor different structural views, and combining complementary tokenizations allows the transformer to leverage distinct signals from each representation.