Dream 7B: Diffusion Large Language Models

TL;DR

Dream 7B introduces a novel diffusion-based large language model that outperforms existing models in various tasks, offering flexible generation and improved planning through innovative training techniques.

Contribution

It is the first large diffusion language model that surpasses autoregressive models in performance and flexibility, with new training methods and capabilities.

Findings

Outperforms existing diffusion language models on multiple tasks.

Demonstrates superior planning and inference flexibility.

Achieves effective training with simple techniques.

Abstract

We introduce Dream 7B, the most powerful open diffusion large language model to date. Unlike autoregressive (AR) models that generate tokens sequentially, Dream 7B employs discrete diffusion modeling to refine sequences in parallel through iterative denoising. Our model consistently outperforms existing diffusion language models on general, mathematical, and coding tasks. Dream 7B demonstrates superior planning abilities and inference flexibility, including arbitrary-order generation, infilling capabilities, and tunable quality-speed trade-offs. These results are achieved through simple yet effective training techniques, including AR-based LLM initialization and context-adaptive token-level noise rescheduling. We release both Dream-Base and Dream-Instruct to facilitate further research in diffusion-based language modeling.