From Sequential to Spatial: Reordering Autoregression for Efficient Visual Generation

TL;DR

RadAR introduces a radial, parallel prediction framework with nested attention to accelerate autoregressive visual generation, maintaining spatial coherence and reducing inference time.

Contribution

The paper proposes RadAR, a novel radial topology-based parallel autoregressive framework with nested attention, enhancing efficiency while preserving visual scene structure.

Findings

RadAR achieves faster inference compared to traditional autoregressive models.

The nested attention mechanism improves prediction consistency.

RadAR maintains high-quality visual generation with increased parallelization.

Abstract

Inspired by the remarkable success of autoregressive models in language modeling, this paradigm has been widely adopted in visual generation. However, the sequential token-by-token decoding mechanism inherent in traditional autoregressive models leads to low inference efficiency.In this paper, we propose RadAR, an efficient and parallelizable framework designed to accelerate autoregressive visual generation while preserving its representational capacity. Our approach is motivated by the observation that visual tokens exhibit strong local dependencies and spatial correlations with their neighbors--a property not fully exploited in standard raster-scan decoding orders. Specifically, we organize the generation process around a radial topology: an initial token is selected as the starting point, and all other tokens are systematically grouped into multiple concentric rings according to their spatial distances from this center. Generation then proceeds in a ring-wise manner, from inner to outer regions, enabling the parallel prediction of all tokens within the same ring. This design not only preserves the structural locality and spatial coherence of visual scenes but also substantially increases parallelization. Furthermore, to address the risk of inconsistent predictions arising from simultaneous token generation with limited context, we introduce a nested attention mechanism. This mechanism dynamically refines implausible outputs during the forward pass, thereby mitigating error accumulation and preventing model collapse. By integrating radial parallel prediction with dynamic output correction, RadAR significantly improves generation efficiency.