The Impossibility Triangle of Long-Context Modeling

TL;DR

This paper proves a fundamental trade-off in long-sequence models, showing that no model can simultaneously be efficient, compact, and have high recall, unifying various architectures under this limitation.

Contribution

The authors formalize a universal trade-off for long-sequence models, proving bounds on recall capacity and classifying existing architectures within this framework.

Findings

Any model satisfying Efficiency and Compactness can recall at most O(poly(d)/log V) key-value pairs.

All 52 analyzed architectures achieve at most two of the three properties in the triangle.

Empirical results confirm that architectures do not surpass the theoretical recall limit.

Abstract

We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall). We formalize this trade-off within an Online Sequence Processor abstraction that unifies Transformers, state space models, linear recurrent networks, and their hybrids. Using the Data Processing Inequality and Fano's Inequality, we prove that any model satisfying Efficiency and Compactness can recall at most O(poly(d)/log V) key-value pairs from a sequence of arbitrary length, where d is the model dimension and V is the vocabulary size. We classify 52 architectures published before March 2026 into the triangle, showing that each achieves at most two of the three properties and that hybrid architectures trace continuous trajectories in the interior. Experiments on synthetic associative recall tasks with five representative architectures validate the theoretical bound: empirical recall capacity lies strictly below the information-theoretic limit, and no architecture escapes the triangle.