CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending

TL;DR

This paper introduces CoCA, a novel attention mechanism that enhances long context window extension in transformers by integrating position embedding with self-attention through a collinear constraint, achieving significant extrapolation improvements.

Contribution

The paper proposes CoCA, a new attention method that seamlessly combines RoPE and self-attention with minimal complexity, enabling transformers to extend context windows up to 32K without fine-tuning.

Findings

CoCA enables GPT models to extend context from 512 to 32K without fine-tuning.

Dropping CoCA in LLaMA-7B achieves 32K context extension within 2K training length.

CoCA performs well as a drop-in replacement for existing transformer models.

Abstract

Self-attention and position embedding are two key modules in transformer-based Large Language Models (LLMs). However, the potential relationship between them is far from well studied, especially for long context window extending. In fact, anomalous behaviors harming long context extrapolation exist between Rotary Position Embedding (RoPE) and vanilla self-attention unveiled by our work. To address this issue, we propose a novel attention mechanism, CoCA (Collinear Constrained Attention). Specifically, we enforce a collinear constraint between $Q$ and $K$ to seamlessly integrate RoPE and self-attention. While only adding minimal computational and spatial complexity, this integration significantly enhances long context window extrapolation ability. We provide an optimized implementation, making it a drop-in replacement for any existing transformer-based models. Extensive experiments show that CoCA performs extraordinarily well in extending context windows. A CoCA-based GPT model, trained with a context length of 512, can seamlessly extend the context window up to 32K (60$\times$), without any fine-tuning. Additionally, by dropping CoCA in LLaMA-7B, we achieve extrapolation up to 32K within only 2K training length. Our code is publicly available at: https://github.com/codefuse-ai/Collinear-Constrained-Attention