Heavy-Tailed Diffusion with Denoising L\'evy Probabilistic Models

TL;DR

This paper introduces Denoising Lévy Probabilistic Models (DLPM), extending diffusion models with heavy-tailed α-stable noise, improving robustness and tail coverage over Gaussian-based models with simpler techniques.

Contribution

The paper proposes a simple extension of DDPM by replacing Gaussian noise with α-stable noise, offering a more flexible and effective heavy-tailed diffusion model.

Findings

Enhanced tail coverage in data distribution

Improved robustness to dataset imbalance

Faster sampling with fewer backward steps

Abstract

Exploring noise distributions beyond Gaussian in diffusion models remains an open challenge. While Gaussian-based models succeed within a unified SDE framework, recent studies suggest that heavy-tailed noise distributions, like $\alpha$-stable distributions, may better handle mode collapse and effectively manage datasets exhibiting class imbalance, heavy tails, or prominent outliers. Recently, Yoon et al.\ (NeurIPS 2023), presented the L\'evy-It\^o model (LIM), directly extending the SDE-based framework to a class of heavy-tailed SDEs, where the injected noise followed an $\alpha$-stable distribution, a rich class of heavy-tailed distributions. However, the LIM framework relies on highly involved mathematical techniques with limited flexibility, potentially hindering broader adoption and further development. In this study, instead of starting from the SDE formulation, we extend the denoising diffusion probabilistic model (DDPM) by replacing the Gaussian noise with $\alpha$-stable noise. By using only elementary proof techniques, the proposed approach, Denoising L\'evy Probabilistic Models (DLPM), boils down to vanilla DDPM with minor modifications. As opposed to the Gaussian case, DLPM and LIM yield different training algorithms and different backward processes, leading to distinct sampling algorithms. These fundamental differences translate favorably for DLPM as compared to LIM: our experiments show improvements in coverage of data distribution tails, better robustness to unbalanced datasets, and improved computation times requiring smaller number of backward steps.