Kernelized Edge Attention: Addressing Semantic Attention Blurring in Temporal Graph Neural Networks

TL;DR

This paper proposes KEAT, a novel attention mechanism for Temporal Graph Neural Networks that distinguishes between node and edge dynamics, improving temporal modeling and interpretability in dynamic graphs.

Contribution

Introduces KEAT, a kernelized edge attention method that preserves the distinct temporal behaviors of nodes and edges in TGNNs, enhancing accuracy and transparency.

Findings

Up to 18% MRR improvement over DyGFormer.

Up to 7% MRR improvement over TGN.

Seamless integration with Transformer and message-passing architectures.

Abstract

Temporal Graph Neural Networks (TGNNs) aim to capture the evolving structure and timing of interactions in dynamic graphs. Although many models incorporate time through encodings or architectural design, they often compute attention over entangled node and edge representations, failing to reflect their distinct temporal behaviors. Node embeddings evolve slowly as they aggregate long-term structural context, while edge features reflect transient, timestamped interactions (e.g. messages, trades, or transactions). This mismatch results in semantic attention blurring, where attention weights cannot distinguish between slowly drifting node states and rapidly changing, information-rich edge interactions. As a result, models struggle to capture fine-grained temporal dependencies and provide limited transparency into how temporal relevance is computed. This paper introduces KEAT (Kernelized Edge Attention for Temporal Graphs), a novel attention formulation that modulates edge features using a family of continuous-time kernels, including Laplacian, RBF, and learnable MLP variant. KEAT preserves the distinct roles of nodes and edges, and integrates seamlessly with both Transformer-style (e.g., DyGFormer) and message-passing (e.g., TGN) architectures. It achieves up to 18% MRR improvement over the recent DyGFormer and 7% over TGN on link prediction tasks, enabling more accurate, interpretable and temporally aware message passing in TGNNs.