From Message-Passing to Linearized Graph Sequence Models

TL;DR

This paper introduces Linearized Graph Sequence Models, recasting message-passing graph computation as sequence modeling to improve long-range information learning and unify architectural design.

Contribution

It provides a new framework that separates processing depth from information propagation, enabling modern sequence modeling techniques to enhance graph learning.

Findings

Improved performance on long-range graph tasks.

Empirical and theoretical analysis of sequence properties for graph learning.

Framework allows treating graph architecture choices as sequence input modeling.

Abstract

Message-passing based approaches form the default backbone of most learning architectures on graph-structured data. However, the rapid progress of modern deep learning architectures in other domains, particularly sequence modeling, raises the question of how graph learning can benefit from these advances. We introduce Linearized Graph Sequence Models, a framework that recasts message-passing graph computation from the perspective of sequence modeling to simplify architectural choices. Our approach systematically separates the computational processing depth from the information propagation depth, allowing core graph architectural decisions to be treated as sequence modeling choices. Specifically, we analyze, both empirically and theoretically, what sequence properties make methods effective for learning and preserving the graph inductive bias. In particular, we validate our findings, demonstrating improved performance on long-range information tasks in graphs. Our findings provide a principled way to integrate modern sequence modeling advances into message-passing based graph learning. Beyond this, our work demonstrates how the separation of processing and information depth can recast central architectural questions as input modeling choices.