Scalable-Softmax Is Superior for Attention

TL;DR

The paper introduces Scalable-Softmax (SSMax), a novel attention mechanism that maintains focus on key information in long contexts, improving language model performance and length generalization.

Contribution

It proposes SSMax as a replacement for Softmax in attention, enhancing long-context modeling and key information prioritization in Transformer-based models.

Findings

SSMax improves long-context attention in language models.

Models with SSMax achieve faster loss reduction during pretraining.

SSMax enhances length generalization even when introduced after pretraining.

Abstract

The maximum element of the vector output by the Softmax function approaches zero as the input vector size increases. Transformer-based language models rely on Softmax to compute attention scores, causing the attention distribution to flatten as the context size grows. This reduces the model's ability to prioritize key information effectively and potentially limits its length generalization. To address this problem, we propose Scalable-Softmax (SSMax), which replaces Softmax in scenarios where the input vector size varies. SSMax can be seamlessly integrated into existing Transformer-based architectures. Experimental results in language modeling show that models using SSMax not only achieve faster loss reduction during pretraining but also significantly improve performance in long contexts and key information retrieval. Furthermore, an analysis of attention scores reveals that SSMax enables the model to focus attention on key information even in long contexts. Additionally, although models that use SSMax from the beginning of pretraining achieve better length generalization, those that have already started pretraining can still gain some of this ability by replacing Softmax in the attention layers with SSMax, either during or after pretraining.