Nexus: Higher-Order Attention Mechanisms in Transformers

TL;DR

Nexus introduces a recursive, higher-order attention mechanism in Transformers that enhances their ability to model complex, multi-hop relationships without increasing parameter count, leading to improved performance.

Contribution

The paper presents Nexus, a novel recursive attention architecture that captures high-order correlations efficiently by dynamically refining queries and keys through nested self-attention loops.

Findings

Nexus outperforms standard Transformers on multiple benchmarks.

The method breaks the linear bottleneck of traditional attention.

Parameter sharing maintains efficiency despite increased expressivity.

Abstract

Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is often limited by a low-rank bottleneck, struggling to capture intricate, multi-hop relationships within a single layer. In this paper, we propose the Nexus, a novel architecture designed to enhance representational power through a recursive framework. Unlike standard approaches that use static linear projections for Queries and Keys, Nexus dynamically refines these representations via nested self-attention mechanisms. Specifically, the Query and Key vectors are themselves outputs of inner attention loops, allowing tokens to aggregate global context and model high-order correlations \textit{prior} to the final attention computation. We enforce a parameter-efficient weight-sharing strategy across recursive steps, ensuring that this enhanced expressivity incurs $\mathcal{O}(1)$ additional parameters. We provide theoretical analysis demonstrating that our method breaks the linear bottleneck of standard attention. Empirically, Nexus outperforms standard Transformers on multiple benchmarks.