Robust Generative Learning with Lipschitz-Regularized $\alpha$-Divergences Allows Minimal Assumptions on Target Distributions

TL;DR

This paper introduces Lipschitz-regularized $oldsymbol{ extalpha}$-divergences as robust objective functionals for generative modeling, enabling stable learning across diverse target distributions with minimal assumptions and providing theoretical guarantees and sample complexity bounds.

Contribution

It establishes the finiteness, existence, and stability of these divergences for generative models, including new bounds and conditions for heavy-tailed and complex distributions.

Findings

Divergences remain finite with minimal assumptions on target distributions.

Stable training of GANs and gradient flows using these divergences.

Numerical experiments demonstrate robustness across challenging scenarios.

Abstract

This paper demonstrates the robustness of Lipschitz-regularized $\alpha$-divergences as objective functionals in generative modeling, showing they enable stable learning across a wide range of target distributions with minimal assumptions. We establish that these divergences remain finite under a mild condition-that the source distribution has a finite first moment-regardless of the properties of the target distribution, making them adaptable to the structure of target distributions. Furthermore, we prove the existence and finiteness of their variational derivatives, which are essential for stable training of generative models such as GANs and gradient flows. For heavy-tailed targets, we derive necessary and sufficient conditions that connect data dimension, $\alpha$, and tail behavior to divergence finiteness, that also provide insights into the selection of suitable $\alpha$'s. We also provide the first sample complexity bounds for empirical estimations of these divergences on unbounded domains. As a byproduct, we obtain the first sample complexity bounds for empirical estimations of these divergences and the Wasserstein-1 metric with group symmetry on unbounded domains. Numerical experiments confirm that generative models leveraging Lipschitz-regularized $\alpha$-divergences can stably learn distributions in various challenging scenarios, including those with heavy tails or complex, low-dimensional, or fractal support, all without any prior knowledge of the structure of target distributions.