Characterizing stable regions in the residual stream of LLMs

TL;DR

This paper identifies stable regions in the residual stream of Transformers, which are linked to semantic distinctions and emerge during training, offering insights into neural network interpretability and dynamics.

Contribution

It introduces the concept of stable regions in the residual stream, revealing their emergence, size, and semantic alignment, advancing understanding of model behavior and interpretability.

Findings

Stable regions are larger than previously studied polytopes.

Regions align with semantic distinctions and prompt clustering.

Activation within a region leads to similar predictions.

Abstract

We identify stable regions in the residual stream of Transformers, where the model's output remains insensitive to small activation changes, but exhibits high sensitivity at region boundaries. These regions emerge during training and become more defined as training progresses or model size increases. The regions appear to be much larger than previously studied polytopes. Our analysis suggests that these stable regions align with semantic distinctions, where similar prompts cluster within regions, and activations from the same region lead to similar next token predictions. This work provides a promising research direction for understanding the complexity of neural networks, shedding light on training dynamics, and advancing interpretability.