Attention layers provably solve single-location regression

TL;DR

This paper introduces the single-location regression task to analyze attention mechanisms, demonstrating their ability to handle sparse token information and internal linear structures through theoretical analysis of a dedicated predictor.

Contribution

It provides a theoretical framework for understanding how attention layers can solve sparse, token-wise regression tasks, including asymptotic optimality and training dynamics analysis.

Findings

The predictor is asymptotically Bayes optimal.

Attention layers can recover latent token positions.

The training dynamics effectively learn the underlying structure.

Abstract

Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.