Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks

TL;DR

This paper analyzes the training dynamics of SGD in sequence single-index models and single-layer attention networks, revealing how sequence length and positional encoding affect learning phases and convergence.

Contribution

It provides a theoretical framework for understanding SGD dynamics in sequential models, extending classical single-index models to attention architectures with explicit phase characterization.

Findings

Two distinct training phases identified: escape and alignment.

Sequence length and positional encoding influence convergence speed.

Closed-form expression for population loss derived.

Abstract

We study the dynamics of stochastic gradient descent (SGD) for a class of sequence models termed Sequence Single-Index (SSI) models, where the target depends on a single direction in input space applied to a sequence of tokens. This setting generalizes classical single-index models to the sequential domain, encompassing simplified one-layer attention architectures. We derive a closed-form expression for the population loss in terms of a pair of sufficient statistics capturing semantic and positional alignment, and characterize the induced high-dimensional SGD dynamics for these coordinates. Our analysis reveals two distinct training phases: escape from uninformative initialization and alignment with the target subspace, and demonstrates how the sequence length and positional encoding influence convergence speed and learning trajectories. These results provide a rigorous and interpretable foundation for understanding how sequential structure in data can be beneficial for learning with attention-based models.