Causal Autoregressive Diffusion Language Model

TL;DR

The paper introduces Causal Autoregressive Diffusion (CARD), a new framework that combines the training efficiency of autoregressive models with the fast inference of diffusion models, enabling efficient and parallel token generation.

Contribution

CARD unifies causal autoregressive training with diffusion inference using a novel causal diffusion reformulation and optimization techniques, improving efficiency and performance.

Findings

Outperforms existing discrete diffusion baselines.

Reduces training latency by 3 times compared to block diffusion.

Achieves ARM-level data efficiency with parallel generation.

Abstract

In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion models. CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass. To address the optimization instability of causal diffusion, we introduce a soft-tailed masking schema to preserve local context and a context-aware reweighting mechanism derived from signal-to-noise principles. This design enables dynamic parallel decoding, where the model leverages KV-caching to adaptively generate variable-length token sequences based on confidence. Empirically, CARD outperforms existing discrete diffusion baselines while reducing training latency by 3 $\times$ compared to block diffusion methods. Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation, establishing a robust paradigm for next-generation efficient LLMs.