Stream-CQSA: Avoiding Out-of-Memory in Attention Computation via Flexible Workload Scheduling

TL;DR

Stream-CQSA introduces a memory-adaptive scheduling framework that enables exact attention computation over billion-token sequences on limited hardware by decomposing attention into independent, schedulable subproblems.

Contribution

The paper presents CQS Divide and Stream-CQSA, novel methods that decompose attention to enable out-of-memory avoidance without approximation or hardware assumptions.

Findings

Exact attention over billion-token sequences achieved on a single GPU.

Memory scaling is predictable and flexible with the proposed framework.

No approximation error introduced in the attention computation.

Abstract

The scalability of long-context large language models is fundamentally limited by the quadratic memory cost of exact self-attention, which often leads to out-of-memory (OOM) failures on modern hardware. Existing methods improve memory efficiency to near-linear complexity, while assuming that the full query, key, and value tensors fit in device memory. In this work, we remove this assumption by introducing CQS Divide, an operation derived from cyclic quorum sets (CQS) theory that decomposes attention into a set of independent subsequence computations whose recomposition yields exactly the same result as full-sequence attention. Exploiting this decomposition, we introduce Stream-CQSA, a memory-adaptive scheduling framework that partitions attention into subproblems that fit within arbitrary memory budgets. This recasts attention from a logically monolithic operation into a collection of schedulable tasks, enabling flexible execution across devices without inter-device communication. Experiments demonstrate predictable memory scaling and show that exact attention over billion-token sequences can be executed on a single GPU via streaming, without changing the underlying mathematical definition of attention or introducing approximation error.