Mirror Diffusion Models

TL;DR

This paper introduces Mirror Diffusion Models (MDMs), a new theoretical framework inspired by mirror Langevin algorithms, to extend diffusion models to constrained discrete and continuous domains like images and text.

Contribution

The paper proposes MDMs, a novel approach for applying diffusion models to constrained domains, with extensions to popular generative tasks such as image and text generation.

Findings

Demonstrates MDMs in simplex diffusion

Extends MDMs to image and text generation

Provides a theoretical foundation for constrained diffusion

Abstract

Diffusion models have successfully been applied to generative tasks in various continuous domains. However, applying diffusion to discrete categorical data remains a non-trivial task. Moreover, generation in continuous domains often requires clipping in practice, which motivates the need for a theoretical framework for adapting diffusion to constrained domains. Inspired by the mirror Langevin algorithm for the constrained sampling problem, in this theoretical report we propose Mirror Diffusion Models (MDMs). We demonstrate MDMs in the context of simplex diffusion and propose natural extensions to popular domains such as image and text generation.