A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers

TL;DR

This paper compares transformer-based and Mamba-based dynamic graph embedding models, showing Mamba's superior efficiency and comparable or better performance in capturing temporal dependencies in evolving networks.

Contribution

It introduces novel Mamba-based models with graph neural networks and provides a comprehensive comparison with transformer approaches, highlighting efficiency and effectiveness.

Findings

Mamba models achieve similar or better link prediction accuracy than transformers.

Mamba models are more computationally efficient, especially on long sequences.

DG-Mamba variants outperform transformers on datasets with high temporal variability.

Abstract

Dynamic graph embedding has emerged as an important technique for modeling complex time-evolving networks across diverse domains. While transformer-based models have shown promise in capturing long-range dependencies in temporal graph data, they face scalability challenges due to quadratic computational complexity. This study presents a comparative analysis of dynamic graph embedding approaches using transformers and the recently proposed Mamba architecture, a state-space model with linear complexity. We introduce three novel models: TransformerG2G augment with graph convolutional networks, \mathcal{DG}-Mamba, and \mathcal{GDG}-Mamba with graph isomorphism network edge convolutions. Our experiments on multiple benchmark datasets demonstrate that Mamba-based models achieve comparable or superior performance to transformer-based approaches in link prediction tasks while offering significant computational efficiency gains on longer sequences. Notably, \mathcal{DG}-Mamba variants consistently outperform transformer-based models on datasets with high temporal variability, such as UCI, Bitcoin, and Reality Mining, while maintaining competitive performance on more stable graphs like SBM. We provide insights into the learned temporal dependencies through analysis of attention weights and state matrices, revealing the models' ability to capture complex temporal patterns. By effectively combining state-space models with graph neural networks, our work addresses key limitations of previous approaches and contributes to the growing body of research on efficient temporal graph representation learning. These findings offer promising directions for scaling dynamic graph embedding to larger, more complex real-world networks, potentially enabling new applications in areas such as social network analysis, financial modeling, and biological system dynamics.