Geometric Attention: A Regime-Explicit Operator Semantics for Transformer Attention

TL;DR

This paper introduces Geometric Attention, a formal framework that explicitly characterizes Transformer attention mechanisms through geometric and algebraic components, enabling principled analysis and extension.

Contribution

It provides a regime-explicit operator semantics for attention, separating invariant structure from modeling choices, and supports diverse attention regimes and extensions.

Findings

Includes softmax kernel as a special case within a broader exponential family.

Provides a canonical low-rank form for interaction components, enabling analysis of attention regimes.

Supports adaptive and staged-depth regimes by extending the carrier update mechanism.

Abstract

Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied). Probe families induce an operational equivalence relation on kernels and therefore a gauge; anchors select representatives relative to that probe. Under a scalar relational-work representation and a multiplicative compositionality law for evidence, the admissible link family is exponential, yielding Gibbs weights; with row anchoring this includes the softmax kernel family as a subregime. After quotienting unary row/column score fields, the remaining interaction component admits a canonical rank-r normal form (Eckart-Young/SVD); dot-product score charts implement the corresponding low-rank interaction regime. Fixing the carrier and extensionalizing the update yields the standard fixed-token Transformer attention operator; allowing carrier updates yields adaptive-carrier and staged-depth regimes. The operator language also supports multihead/mixed kernels, plan-based anchors (e.g., entropic OT/Sinkhorn), and unary operators (e.g., FFN-style fields) as explicit regime choices. This separates invariant structure from modeling choice, enabling principled comparison and extension of attention mechanisms, and attention-based architectures.