Improving global awareness of linkset predictions using Cross-Attentive Modulation tokens

TL;DR

This paper introduces Cross-Attentive Modulation tokens to enhance global awareness in link prediction models, improving their ability to capture high-level graph features and outperforming baseline methods.

Contribution

The paper proposes CAM tokens that gather information via cross-attention to improve graph neural network models' global awareness for link prediction tasks.

Findings

CAM tokens improve model performance on link prediction benchmarks.

CAM-enhanced models outperform baseline models with statistical graph attributes.

The approach is effective in both simple attention models and graph transformers.

Abstract

This work introduces Cross-Attentive Modulation (CAM) tokens, which are tokens whose initial value is learned, gather information through cross-attention, and modulate the nodes and edges accordingly. These tokens are meant to improve the global awareness of link predictions models which, based on graph neural networks, can struggle to capture graph-level features. This lack of ability to feature high level representations is particularly limiting when predicting multiple or entire sets of links. We implement CAM tokens in a simple attention-based link prediction model and in a graph transformer, which we also use in a denoising diffusion framework. A brief introduction to our toy datasets will then be followed by benchmarks which prove that CAM token improve the performance of the model they supplement and outperform a baseline with diverse statistical graph attributes.