Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff

TL;DR

This paper investigates why two-stream attention is essential in any-order autoregressive models, revealing it helps balance competing semantic and structural information during sequence generation.

Contribution

The paper identifies a structural-semantic tradeoff in any-order generation and proposes Decoupled RoPE to address it, clarifying the role of two-stream attention.

Findings

Two-stream attention helps mitigate the structural-semantic tradeoff.

Decoupled RoPE improves performance at short sequence lengths.

Performance degrades as sequence length increases and tradeoff diverges.

Abstract

Any-order autoregressive models (AO-ARMs) offer a promising path toward efficient masked diffusion by enabling native key-value caching, but competitive performance has so far required two-stream attention, typically motivated as a means of decoupling token content from position. In this work, we argue that two-stream attention may be serving a more subtle role. We identify a structural-semantic tradeoff in any-order generation: the hidden representation at each step must simultaneously attend to semantically informative tokens for prediction and structurally recent tokens for summarization, objectives that compete for attention capacity in a single stream but can specialize across two streams. To isolate this tradeoff from position-content separation, we propose Decoupled RoPE, a modification to rotary position embeddings that provides target position information without revealing target content. Decoupled RoPE performs competitively at short sequence lengths--where semantic and structural proximity coincide--but degrades as sequence length increases and the two orderings diverge. These results suggest that the success of two-stream attention stems not merely from separating position from content, but from circumventing the deeper structural-semantic tradeoff inherent to any-order generation.