Towards Latent Diffusion Suitable For Text

TL;DR

This paper introduces Neural Flow Diffusion Models for language generation, which adapt continuous diffusion techniques to discrete language data, improving sampling speed and quality while reducing likelihood gaps compared to autoregressive models.

Contribution

It extends NFDM to enable continuous diffusion in discrete spaces, providing a new approach for efficient and high-quality language modeling.

Findings

Reduces likelihood gap with autoregressive models

Achieves sample quality comparable to previous latent diffusion models

Substantially improves sampling efficiency

Abstract

Language diffusion models aim to improve sampling speed and coherence over autoregressive LLMs. We introduce Neural Flow Diffusion Models for language generation, an extension of NFDM that enables the straightforward application of continuous diffusion models to discrete state spaces. NFDM learns a multivariate forward process from the data, ensuring that the forward process and generative trajectory are a good fit for language modeling. Our model substantially reduces the likelihood gap with autoregressive models of the same size, while achieving sample quality comparable to that of previous latent diffusion models.