Heavy-Tailed Diffusion Models

TL;DR

This paper introduces a novel heavy-tailed diffusion framework using Student-t distributions, enhancing the modeling of rare and extreme events in data, with practical extensions and improved performance on weather datasets.

Contribution

It develops a heavy-tail diffusion framework with a tailored perturbation kernel and training objective, compatible with existing models, for better heavy-tail behavior modeling.

Findings

Outperforms standard models in heavy-tail estimation

Effectively models rare and extreme weather events

Requires minimal modifications to existing diffusion models

Abstract

Diffusion models achieve state-of-the-art generation quality across many applications, but their ability to capture rare or extreme events in heavy-tailed distributions remains unclear. In this work, we show that traditional diffusion and flow-matching models with standard Gaussian priors fail to capture heavy-tailed behavior. We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions. We develop a tailored perturbation kernel and derive the denoising posterior based on the conditional Student-t distribution for the backward process. Inspired by $\gamma$-divergence for heavy-tailed distributions, we derive a training objective for heavy-tailed denoisers. The resulting framework introduces controllable tail generation using only a single scalar hyperparameter, making it easily tunable for diverse real-world distributions. As specific instantiations of our framework, we introduce t-EDM and t-Flow, extensions of existing diffusion and flow models that employ a Student-t prior. Remarkably, our approach is readily compatible with standard Gaussian diffusion models and requires only minimal code changes. Empirically, we show that our t-EDM and t-Flow outperform standard diffusion models in heavy-tail estimation on high-resolution weather datasets in which generating rare and extreme events is crucial.