Scaled Dot-Product Attention implements projection of inputs onto a common surface

TL;DR

This paper reinterprets scaled dot-product attention as a projection onto a common surface, revealing nonlinear, time-dependent dependencies in inputs and suggesting new extensions and interpretations for language and time-series processing.

Contribution

It provides a new mathematical formulation of SDPA as a projection, offering insights into its nonlinear and context-dependent nature, and proposes potential extensions.

Findings

SDPA can be expressed as a projection onto a common surface.

This reinterpretation reveals nonlinear, time-dependent input dependencies.

The new view suggests extensions and different roles for SDPA in language and time-series analysis.

Abstract

Scaled dot-product attention (SDPA) is a fundamental component responsible for the success of large-language models and other nonlinear signal processing applications. The rationale for SDPA has been based upon "query, key, value" concepts borrowed from database theory, but these concepts are difficult to reconcile with standard methods in mathematical signal processing. We show that SDPA can be rewritten in a different but mathematically equivalent form as a projection of the input vectors onto a common surface determined by the inputs themselves. Therefore SDPA discovers nonlinear dependencies in the input that are time-dependent and context-dependent. The rewritten form of SDPA permits increased speed of both feedforward and learning algorithms, but more importantly suggests potential extensions. In the context of language, we re-interpret the role of SDPA as finding a time-dependent contextual meaning determined by the surface on which the set of input vectors lies. Input token embeddings are then modified by the local context surface. This interpretation differs substantially from the concept of "self-attention", and provides a strong justification for the use of SDPA for time-series data with time-varying local nonlinear dependencies.