Parametric Adversarial Divergences are Good Losses for Generative Modeling

TL;DR

This paper argues that parametric adversarial divergences, despite being approximations, possess unique properties that make them more suitable than nonparametric divergences for learning high-dimensional distributions in generative modeling.

Contribution

The paper demonstrates that parametric divergences are sensitive to specific distribution aspects, offering advantages over nonparametric divergences for high-dimensional generative tasks.

Findings

Parametric divergences focus on certain distribution moments.

They can be more aligned with sample quality than nonparametric divergences.

Parametric divergences provide intuitive alternatives to mutual information.

Abstract

Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator assumption. In this position paper, we argue that despite being "non-optimal", parametric divergences have distinct properties from their nonparametric counterparts which can make them more suitable for learning high-dimensional distributions. A key property is that parametric divergences are only sensitive to certain aspects/moments of the distribution, which depend on the architecture of the discriminator and the loss it was trained with. In contrast, nonparametric divergences such as the Kullback-Leibler divergence are sensitive to moments ignored by the discriminator, but they do not necessarily correlate with sample quality (Theis et al., 2016). Similarly, we show that mutual information can lead to unintuitive interpretations, and explore more intuitive alternatives based on parametric divergences. We conclude that parametric divergences are a flexible framework for defining statistical quantities relevant to a specific modeling task.