Striped Attention: Faster Ring Attention for Causal Transformers

TL;DR

This paper introduces Striped Attention, an extension of Ring Attention for causal transformers that improves workload balance and significantly increases training throughput on large sequence lengths.

Contribution

We propose Striped Attention, a simple modification to Ring Attention that balances workloads and enhances performance for causal transformer training.

Findings

Achieved up to 1.45x throughput improvement on GPUs.

Achieved 1.65x speedup on TPUv4s at longer sequences.

Demonstrated effectiveness on sequence lengths up to 786k.

Abstract

To help address the growing demand for ever-longer sequence lengths in transformer models, Liu et al. recently proposed Ring Attention, an exact attention algorithm capable of overcoming per-device memory bottle- necks by distributing self-attention across multiple devices. In this paper, we study the performance characteristics of Ring Attention in the important special case of causal transformer models, and identify a key workload imbal- ance due to triangular structure of causal attention computations. We propose a simple extension to Ring Attention, which we call Striped Attention to fix this imbalance. Instead of devices having contiguous subsequences, each device has a subset of tokens distributed uniformly throughout the sequence, which we demonstrate leads to more even workloads. In experiments running Striped Attention on A100 GPUs and TPUv4s, we are able to achieve up to 1.45x end-to-end throughput improvements over the original Ring Attention algorithm on causal transformer training at a sequence length of 256k. Furthermore, on 16 TPUv4 chips, we were able to achieve 1.65x speedups at sequence lengths of 786k. We release the code for our experiments as open source