Conditional [MASK] Discrete Diffusion Language Model

TL;DR

This paper introduces Diffusion-EAGS, a novel diffusion-based language model that combines conditional masked language models with entropy-adaptive techniques to improve diversity and controllability in non-autoregressive text generation.

Contribution

It proposes a new framework integrating conditional masked language models into diffusion models via a theoretical conditional Markov Random Field approach, with entropy-adaptive sampling methods.

Findings

Outperforms baseline models in quality-diversity tradeoff

Achieves superior diversity in generated text

Demonstrates effectiveness in non-autoregressive generation

Abstract

Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropy-based noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.