How Smoothing is N-simplicial Attention?

TL;DR

This paper introduces N-simplicial attention to incorporate higher-order interactions in message-passing mechanisms, adapting it for Rotary Position Embeddings and analyzing its smoothing properties and over-smoothing issues.

Contribution

It presents a novel N-simplicial attention mechanism with a cost-effective simplex selection and adapts it for RoPE, advancing higher-order interaction modeling in graph neural networks.

Findings

Derived a Lipschitz upper-bound for smoothing effects.

Demonstrated that N-simplicial attention suffers from over-smoothing.

Proposed a simplex selection method to focus on task-sensitive interactions.

Abstract

Going from pure Multilayer Perceptron (MLP) to a learnable graph message-passing mechanism at each layer has been foundational to state-of-the-art results, despite the computational trade-off (e.g. GATs or Transformers). To go a step further, in this work, we introduce N-simplicial attention, going from pairwise token similarity to higher-order interactions, and adapt it for Rotary Position Embeddings (RoPE). To help manage the increased complexity, we propose a cost-effective simplex selection enabling the model to focus its computation load onto the more task-sensitive interactions. Beyond these core mechanisms, we study how smoothing N-simplicial attention is by deriving a Lipschitz upper-bound and by demonstrating that by itself it also suffers from over-smoothing, despite opening the attention message-passing to higher-order interactions.