G2rammar: Bilingual Grammar Modeling for Enhanced Text-attributed Graph Learning

TL;DR

G2rammar introduces a bilingual grammar framework that explicitly encodes structural and semantic roles in text-attributed graphs, significantly improving language models' ability to understand complex graph topologies.

Contribution

It presents a novel bilingual grammar encoding approach for graphs, combining structural and semantic grammar to enhance graph learning with language models.

Findings

Outperforms baseline models on real-world datasets.

Provides effective encoding of graph topological roles.

Enhances language models' reasoning about graph structures.

Abstract

Text-attributed graphs require models to effectively integrate both structural topology and semantic content. Recent approaches apply large language models to graphs by linearizing structures into token sequences through random walks. These methods create concise graph vocabularies to replace verbose natural language descriptions. However, they overlook a critical component that makes language expressive: grammar. In natural language, grammar assigns syntactic roles to words and defines their functions within sentences. Similarly, nodes in graphs play distinct structural roles as hubs, bridges, or peripheral members. Current graph language methods provide tokens without grammatical annotations to indicate these structural or semantic roles. This absence limits language models' ability to reason about graph topology effectively. We propose \textbf{G2rammar}, a bilingual grammar framework that explicitly encodes both structural and semantic grammar for text-attributed graphs. Structural grammar characterizes topological roles through centrality and neighborhood patterns. Semantic grammar captures content relationships through textual informativity. The framework implements two-stage learning with structural grammar pre-training followed by semantic grammar fine-tuning. Extensive experiments on real-world datasets demonstrate that G2rammar consistently outperforms competitive baselines by providing language models with the grammatical context needed to understand graph structures.