Attention Projection Mixing with Exogenous Anchors

TL;DR

ExoFormer introduces external exogenous anchors and a normalized mixing framework to improve attention reuse, leading to better performance and efficiency in transformer models.

Contribution

The paper proposes ExoFormer, a novel approach that learns external anchor projections and a unified mixing framework, overcoming internal-anchor limitations and enhancing model performance.

Findings

ExoFormer variants outperform internal-anchor models.

Dynamic ExoFormer achieves 1.5x accuracy points with fewer tokens.

Normalized anchor sources are crucial for stable reuse.

Abstract

Cross-layer reuse of early attention projections can improve optimization and data efficiency, but it creates a structural conflict: the first layer must simultaneously act as a stable, reusable anchor for all deeper layers and as an effective computational block. We demonstrate that this tension constrains the performance of internal-anchor designs. We propose ExoFormer, which resolves the conflict by learning exogenous anchor projections outside the sequential layer stack. We introduce a unified normalized mixing framework that mixes queries, keys, values, and gate logits using learnable coefficients (exploring coefficient granularities: elementwise, headwise, and scalar), and we show that normalizing anchor sources is key to stable reuse. ExoFormer variants consistently outperform their internal-anchor counterparts, and the dynamic variant yields 1.5x downstream accuracy points while matching validation loss using 1.5x fewer tokens than Gated Attention. We explain this efficacy via an Offloading Hypothesis: external anchors preserve essential token identity, allowing layers to specialize exclusively in feature transformation. We release code and models to facilitate future research.