Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning

TL;DR

This paper introduces GTLM, a parameter-efficient LLM architecture that natively processes graph structures, outperforming larger models on graph reasoning benchmarks by integrating graph-aware attention.

Contribution

GTLM enables pretrained LLMs to directly process graph topologies with minimal parameter overhead, preserving permutation equivariance and backward compatibility.

Findings

GTLM matches or exceeds larger models on graph benchmarks.

GTLM's attention heads learn message passing behavior.

GTLM significantly outperforms baselines on GraphQA.

Abstract

Using Large Language Models (LLMs) to process graph-structured data is an active research area, yet current state-of-the-art approaches typically rely on multi-step pipelines with Graph Neural Network (GNN) encoders that compress rich textual attributes into solitary tokens, creating a significant semantic bottleneck. In this paper, we introduce the Graph Transformer Language Model (GTLM), a novel architecture that enables pretrained LLMs to natively process graph topologies while entirely eliminating this compressive bottleneck. GTLM is exceptionally parameter-efficient: by injecting graph-aware attention biases directly into the LLM's attention modules, it introduces only 0.015% additional parameters relative to the base model. We theoretically prove that our bidirectional attention prefix preserves node permutation equivariance while maintaining exact backward compatibility with the pretrained base model. Extensive evaluations demonstrate that a 1B-parameter GTLM matches or exceeds the performance of 7B-parameter state-of-the-art models on standard Text-Attributed Graph benchmarks, while significantly surpassing baselines on GraphQA. Finally, we demonstrate that GTLM attention heads implicitly learn to simulate message passing, explaining its superior performance on algorithmic tasks. This paradigm shift enables true algorithmic reasoning within LLMs and provides a scalable foundation for next-generation GraphRAG and relational deep learning.