SVDformer: Direction-Aware Spectral Graph Embedding Learning via SVD and Transformer

TL;DR

SVDformer introduces a spectral-Transformer framework that effectively captures directionality and global structure in directed graphs, outperforming existing methods in node classification tasks.

Contribution

The paper presents SVDformer, a novel spectral-Transformer model that jointly models spectral components and edge directionality without spectral kernels, advancing directed graph representation learning.

Findings

Outperforms state-of-the-art GNNs on six directed graph benchmarks.

Effectively models multi-scale interactions between edge patterns.

Enhances spectral components through multi-head self-attention.

Abstract

Directed graphs are widely used to model asymmetric relationships in real-world systems. However, existing directed graph neural networks often struggle to jointly capture directional semantics and global structural patterns due to their isotropic aggregation mechanisms and localized filtering mechanisms. To address this limitation, this paper proposes SVDformer, a novel framework that synergizes SVD and Transformer architecture for direction-aware graph representation learning. SVDformer first refines singular value embeddings through multi-head self-attention, adaptively enhancing critical spectral components while suppressing high-frequency noise. This enables learnable low-pass/high-pass graph filtering without requiring spectral kernels. Furthermore, by treating singular vectors as directional projection bases and singular values as scaling factors, SVDformer uses the Transformer to model multi-scale interactions between incoming/outgoing edge patterns through attention weights, thereby explicitly preserving edge directionality during feature propagation. Extensive experiments on six directed graph benchmarks demonstrate that SVDformer consistently outperforms state-of-the-art GNNs and direction-aware baselines on node classification tasks, establishing a new paradigm for learning representations on directed graphs.