Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training

TL;DR

This paper introduces a rank-aware concentration inequality for attention scores in transformers, enabling more precise overflow risk assessment and stable low-precision training, especially for FP8 precision.

Contribution

It derives a novel rank-aware bound on attention score magnitudes and applies it to develop geometry-aware scale factors for overflow prevention in low-precision transformer training.

Findings

Tighter concentration bounds for attention scores in low-rank settings

Effective overflow prevention in FP8 training across large models

Maintains accuracy while eliminating overflows in practical scenarios

Abstract

Attention scores in transformers are bilinear forms $S_{ij} = x_i^\top M x_j / \sqrt{d_h}$ whose maximum magnitude governs overflow risk in low-precision training. We derive a \emph{rank-aware concentration inequality}: when the interaction matrix $M = W^Q W^{K\top}$ has rank $r \ll d$, tail probabilities for $\max_{i,j}|S_{ij}|$ decay as $\exp(-d^{2}\alpha^{2}/(\gamma r))$ rather than $\exp(-d\alpha^{2})$, where $\gamma > 1$ is a typicality parameter. For transformer attention where $r = d_h$, this yields $8$--$28\times$ tighter concentration than rank-agnostic bounds in modern architectures. We apply this result to FP8 training, deriving \emph{geometry-aware scale factors} that provide principled overflow guarantees without observing activations. The method computes per-layer scales from the spectral norm $\|W^Q W^{K\top}\|_2$ via implicit power iteration, includes a grouped query attention formulation that avoids key expansion, and remains compatible with fused attention kernels. Across GPT-2 XL to Llama-2-70B, geometry-aware scaling eliminates overflows in transient scenarios where delayed scaling fails, while achieving comparable downstream MMLU accuracy.