Why Gaussian Diffusion Models Fail on Discrete Data?

TL;DR

Gaussian diffusion models struggle with discrete data due to multimodal density issues, but combining heuristics like self-conditioning and q-sampling improves their performance.

Contribution

The paper identifies the cause of diffusion model failures on discrete data and proposes combined heuristics to enhance sampling quality.

Findings

Multimodal density causes diffusion sampling failures on discrete data.

Self-conditioning and q-sampling heuristics mitigate multimodal issues.

Combining these methods improves generation quality across various domains.

Abstract

Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and proteins.