LINA: Linear Autoregressive Image Generative Models with Continuous Tokens

TL;DR

LINA introduces a compute-efficient linear attention-based model for text-to-image synthesis, achieving high-quality 1024x1024 images with significantly reduced computational costs and competitive benchmark performance.

Contribution

The paper designs and evaluates a novel linear attention mechanism with gating and convolutional augmentations, enabling a simple, efficient, and high-fidelity T2I model called LINA.

Findings

Division-based normalization scales better for generative transformers.

Convolutional locality modeling improves autoregressive image generation.

LINA reduces FLOPs by 61% compared to softmax attention and achieves competitive benchmark results.

Abstract

Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis, but they suffer from high computational cost. We study how to design compute-efficient linear attention within this framework. Specifically, we conduct a systematic empirical analysis of scaling behavior with respect to parameter counts under different design choices, focusing on (1) normalization paradigms in linear attention (division-based vs. subtraction-based) and (2) depthwise convolution for locality augmentation. Our results show that although subtraction-based normalization is effective for image classification, division-based normalization scales better for linear generative transformers. In addition, incorporating convolution for locality modeling plays a crucial role in autoregressive generation, consistent with findings in diffusion models. We further extend gating mechanisms, commonly used in causal linear attention, to the bidirectional setting and propose a KV gate. By introducing data-independent learnable parameters to the key and value states, the KV gate assigns token-wise memory weights, enabling flexible memory management similar to forget gates in language models. Based on these findings, we present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions. LINA achieves competitive performance on both class-conditional and T2I benchmarks, obtaining 2.18 FID on ImageNet (about 1.4B parameters) and 0.74 on GenEval (about 1.5B parameters). A single linear attention module reduces FLOPs by about 61 percent compared to softmax attention. Code and models are available at: https://github.com/techmonsterwang/LINA.