Differences in Text Generated by Diffusion and Autoregressive Language Models

TL;DR

This paper compares diffusion and autoregressive language models, revealing that diffusion models produce more coherent and diverse text due to their training objectives and decoding strategies, with implications for future model design.

Contribution

It provides an empirical and theoretical analysis of the intrinsic differences between diffusion and autoregressive language models, highlighting the roles of training objectives and decoding algorithms.

Findings

Diffusion models have lower n-gram entropy than autoregressive models.

Diffusion models exhibit higher semantic coherence and diversity.

Decoding algorithms mainly cause the entropy reduction in diffusion models.

Abstract

Diffusion language models (DLMs) are promising alternatives to autoregressive language models (ARMs), yet the intrinsic differences in their generated text remain underexplored. We first find empirically that off-the-shelf DLMs exhibit lower $n$-gram entropy, higher semantic coherence, and higher semantic diversity. To understand the cause, we conduct controlled experiments that decouple the effects of training objectives and decoding algorithms. Results suggest that the DLM training objective contributes to the increases in semantic coherence and semantic diversity, but has a minor influence on entropy. These differences are primarily driven by the bidirectional context; other components in the training objective, such as input masking, label masking, and the weighting function, have a much weaker influence. Further, our experiments demonstrate that the reduction in entropy stems from DLMs' decoding algorithms, particularly confidence-based remasking strategies. We provide a theoretical understanding for this entropy reduction phenomenon. Together, our work uncovers key mechanisms underlying the differences between DLMs and ARMs in text generation, and informs future design of training objectives and decoding algorithms in DLMs.