Global-Lens Transformers: Adaptive Token Mixing for Dynamic Link Prediction

TL;DR

This paper introduces GLFormer, an attention-free Transformer variant for dynamic graphs that uses adaptive token mixing and hierarchical aggregation to efficiently model evolving relationships, achieving state-of-the-art results.

Contribution

Proposes GLFormer, a novel attention-free Transformer architecture with adaptive token mixing and hierarchical modules for dynamic graph modeling, challenging the necessity of self-attention.

Findings

GLFormer achieves state-of-the-art performance on six dynamic graph benchmarks.

The attention-free design improves efficiency while maintaining or surpassing Transformer baselines.

Hierarchical aggregation captures long-term dependencies effectively.

Abstract

Dynamic graph learning plays a pivotal role in modeling evolving relationships over time, especially for temporal link prediction tasks in domains such as traffic systems, social networks, and recommendation platforms. While Transformer-based models have demonstrated strong performance by capturing long-range temporal dependencies, their reliance on self-attention results in quadratic complexity with respect to sequence length, limiting scalability on high-frequency or large-scale graphs. In this work, we revisit the necessity of self-attention in dynamic graph modeling. Inspired by recent findings that attribute the success of Transformers more to their architectural design than attention itself, we propose GLFormer, a novel attention-free Transformer-style framework for dynamic graphs. GLFormer introduces an adaptive token mixer that performs context-aware local aggregation based on interaction order and time intervals. To capture long-term dependencies, we further design a hierarchical aggregation module that expands the temporal receptive field by stacking local token mixers across layers. Experiments on six widely-used dynamic graph benchmarks show that GLFormer achieves SOTA performance, which reveals that attention-free architectures can match or surpass Transformer baselines in dynamic graph settings with significantly improved efficiency.