Limitations of Normalization in Attention Mechanism

TL;DR

This paper analyzes the limitations of normalization in attention mechanisms, revealing how it affects token selection and model sensitivity, and provides both theoretical insights and empirical validation using GPT-2.

Contribution

It offers a theoretical framework for understanding normalization effects in attention, highlighting issues with token distinguishability and training sensitivity, and suggests directions for more robust attention designs.

Findings

Increased selected tokens reduce token distinguishability.

Softmax normalization causes gradient sensitivity issues during training.

Model tends toward uniform token selection as token count grows.

Abstract

This paper investigates the limitations of the normalization in attention mechanisms. We begin with a theoretical framework that enables the identification of the model's selective ability and the geometric separation involved in token selection. Our analysis includes explicit bounds on distances and separation criteria for token vectors under softmax scaling. Through experiments with pre-trained GPT-2 model, we empirically validate our theoretical results and analyze key behaviors of the attention mechanism. Notably, we demonstrate that as the number of selected tokens increases, the model's ability to distinguish informative tokens declines, often converging toward a uniform selection pattern. We also show that gradient sensitivity under softmax normalization presents challenges during training, especially at low temperature settings. These findings advance current understanding of softmax-based attention mechanism and motivate the need for more robust normalization and selection strategies in future attention architectures.