ARFlow: Autoregressive Flow with Hybrid Linear Attention

TL;DR

ARFlow integrates autoregressive modeling with flow models and a hybrid linear attention mechanism to improve image generation quality and capture long-range dependencies more effectively.

Contribution

This paper introduces ARFlow, a novel flow model that combines autoregressive training and a custom hybrid linear attention for enhanced image generation.

Findings

Achieves 6.63 FID on ImageNet 256x256 without guidance.

Outperforms previous flow-based models like SiT.

Demonstrates effectiveness of chunk-wise attention and autoregressive training.

Abstract

Flow models are effective at progressively generating realistic images, but they generally struggle to capture long-range dependencies during the generation process as they compress all the information from previous time steps into a single corrupted image. To address this limitation, we propose integrating autoregressive modeling -- known for its excellence in modeling complex, high-dimensional joint probability distributions -- into flow models. During training, at each step, we construct causally-ordered sequences by sampling multiple images from the same semantic category and applying different levels of noise, where images with higher noise levels serve as causal predecessors to those with lower noise levels. This design enables the model to learn broader category-level variations while maintaining proper causal relationships in the flow process. During generation, the model autoregressively conditions the previously generated images from earlier denoising steps, forming a contextual and coherent generation trajectory. Additionally, we design a customized hybrid linear attention mechanism tailored to our modeling approach to enhance computational efficiency. Our approach, termed ARFlow, achieves 6.63 FID scores on ImageNet at 256 * 256 without classifier-free guidance, reaching 1.96 FID with classifier-free guidance 1.5, outperforming the previous flow-based model SiT's 2.06 FID. Extensive ablation studies demonstrate the effectiveness of our modeling strategy and chunk-wise attention design.