When Graph Language Models Go Beyond Memorization

TL;DR

This paper introduces a diagnostic protocol to distinguish whether graph language models learn structural regularities or simply memorize training graphs, revealing that large-scale models can acquire meaningful structural knowledge beyond memorization.

Contribution

The authors develop a calibrated diagnostic framework combining subgraph mining and frequency stratification to disentangle memorization from structural learning in graph language models.

Findings

Large-scale models show high subgraph-rank correlation, indicating structural learning.

Memorization drops sharply at scale, while structural alignment persists.

High-frequency patterns are well reproduced, rare patterns less so.

Abstract

It remains unclear whether graph language models learn structural regularities or merely memorize training graphs; this cannot be resolved by current aggregate fidelity metrics alone. We develop a calibrated diagnostic protocol that combines frequent subgraph mining, a graph-level bootstrap baseline, and three-level frequency stratification to disentangle memorization from structural alignment. Using this framework, we show that graph language models can acquire structural regularities beyond memorization at scale, primarily in the high-frequency regime. This is supported by the following empirical evidence: On five TU benchmarks, LLaMA-style graph language models reach high subgraph-rank correlation, yet their alignment is matched or exceeded by the memorization bootstrap in most cases. At small scale, under our bootstrap diagnostic, fidelity is largely indistinguishable from verbatim recall. In contrast, at large scale with 3.75M graphs, verbatim memorization drops sharply while rank correlation remains near ceiling. Crucially, in a separate fixed-subsample analysis, frequent subgraph mining restricted to the novel-only subset closely tracks the corresponding all-generation Spearman correlation, providing evidence that the alignment is not driven solely by verbatim recall. Across all scales, high-frequency patterns are well reproduced, while rare patterns remain poorly covered, and this deficit narrows only marginally as capacity increases. We observe the same scale-dependent crossover under two distinct graph serializations (canonical DFS code and action sequences), providing evidence of robustness in our analysis.