On the Hardness of Sampling from Mixture Distributions via Langevin Dynamics

TL;DR

This paper investigates the challenges of using Langevin Dynamics to sample from mixture distributions, revealing its limitations in high dimensions and proposing a chained variant that improves convergence, supported by theoretical analysis and experiments.

Contribution

The paper provides the first theoretical analysis of Langevin Dynamics on mixture distributions and introduces Chained-LD to enhance sampling efficiency.

Findings

Langevin Dynamics may fail to find all mixture components in high dimensions.

Chained-LD converges faster by sequentially sampling data patches.

Experimental results validate theoretical iteration complexity bounds.

Abstract

The Langevin Dynamics (LD), which aims to sample from a probability distribution using its score function, has been widely used for analyzing and developing score-based generative modeling algorithms. While the convergence behavior of LD in sampling from a uni-modal distribution has been extensively studied in the literature, the analysis of LD under a mixture distribution with distinct modes remains underexplored in the literature. In this work, we analyze LD in sampling from a mixture distribution and theoretically study its convergence properties. Our theoretical results indicate that for general mixture distributions of sub-Gaussian components, LD could fail in finding all the components within a sub-exponential number of steps in the data dimension. Following our result on the complexity of LD in sampling from high-dimensional variables, we propose Chained Langevin Dynamics (Chained-LD), which divides the data vector into patches of smaller sizes and generates every patch sequentially conditioned on the previous patches. Our theoretical analysis of Chained-LD indicates its faster convergence speed to the components of a mixture distribution. We present the results of several numerical experiments on synthetic and real image datasets, validating our theoretical results on the iteration complexities of sample generation from mixture distributions using the vanilla and chained LD algorithms.