Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

TL;DR

This paper introduces Deletion-Insertion Diffusion models (DID) for language modeling, which improve efficiency and flexibility over masked diffusion models by using token deletion and insertion as diffusion processes, enabling variable-length sequences and self-correction.

Contribution

The paper proposes a novel diffusion framework based on token deletion and insertion, replacing masking, with a score-based training method and dynamic programming for efficient training, improving over existing models.

Findings

DID outperforms baselines in modeling performance and sampling quality.

DID achieves faster training and inference speeds.

DID supports variable-length sequences without padding.

Abstract

While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Insertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead in MDLMs: the computations on non-informative 1) <MASK> tokens inherent to the paradigm, and 2) <PAD> tokens introduced in variable-length settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design a score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which we efficiently solve via a parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.