Generative Modelling with High-Order Langevin Dynamics

TL;DR

This paper introduces HOLD, a high-order Langevin dynamics method that enhances diffusion models by modeling position, velocity, and acceleration, leading to faster and higher-quality data generation, especially in image synthesis.

Contribution

The paper proposes a novel high-order Langevin dynamics approach that improves data generation speed and quality by augmenting existing SDEs with higher-order derivatives.

Findings

Achieves state-of-the-art FID of 1.85 on CIFAR-10

Reduces mixing time by two orders of magnitude

Significantly improves image generation quality and speed

Abstract

Diffusion generative modelling (DGM) based on stochastic differential equations (SDEs) with score matching has achieved unprecedented results in data generation. In this paper, we propose a novel fast high-quality generative modelling method based on high-order Langevin dynamics (HOLD) with score matching. This motive is proved by third-order Langevin dynamics. By augmenting the previous SDEs, e.g. variance exploding or variance preserving SDEs for single-data variable processes, HOLD can simultaneously model position, velocity, and acceleration, thereby improving the quality and speed of the data generation at the same time. HOLD is composed of one Ornstein-Uhlenbeck process and two Hamiltonians, which reduce the mixing time by two orders of magnitude. Empirical experiments for unconditional image generation on the public data set CIFAR-10 and CelebA-HQ show that the effect is significant in both Frechet inception distance (FID) and negative log-likelihood, and achieves the state-of-the-art FID of 1.85 on CIFAR-10.