NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models

TL;DR

NAG introduces a unified, encoder-free architecture that integrates graph processing directly into language models using self-attention, simplifying text-graph modeling and improving comprehension without external encoders.

Contribution

The paper proposes NAG, a novel framework that internalizes graph processing within language models, eliminating the need for external GNNs and enabling more coherent text-graph understanding.

Findings

NAG outperforms traditional methods on various graph tasks.

NAG maintains linguistic capabilities while enhancing structural understanding.

Efficient implementations like NAG-Zero and NAG-LoRA demonstrate versatility.

Abstract

Prevailing methods for integrating graphs into Language Models (LMs) typically rely on a segregated architecture: external Graph Neural Networks (GNNs) encode structural topology, while LMs process textual semantics. We argue this approach is suboptimal for text-graphs: it creates a conceptually disjointed interaction paradigm. By segregating structural encoding from semantic processing, these systems must perform a complex implicit alignment between abstract graph tokens and concrete textual elements. Challenging the necessity of external encoders, we propose NAG (Native Architecture for Graphs), a unified framework that internalizes graph processing within the LM's native manifold. Instead of bridging disparate embedding spaces, NAG repurposes the self-attention mechanism to enforce topological dependencies and recalibrates positional IDs to ensure structural equivalence. This allows the model to harness its intrinsic linguistic capability to simultaneously comprehend node and edge content alongside structural topology. We introduce two efficient implementations: NAG-Zero for absolute preservation of the base model's linguistic capabilities, and NAG-LoRA for enhanced structural adaptation. Experiments across diverse graph tasks validate that NAG achieves robust graph comprehension without the overhead of external encoders, offering a simpler, more coherent paradigm for text-graph modeling.