Spread Divergence

TL;DR

This paper introduces Spread Divergence, a new method for comparing distributions with different supports or undefined densities, and demonstrates its application in training implicit generative models.

Contribution

It defines Spread Divergence, provides conditions for its existence, and shows how to optimize and apply it in generative modeling tasks.

Findings

Successfully trained implicit generative models using Spread Divergence

Demonstrated effectiveness on linear and non-linear models

Provided theoretical conditions for divergence existence

Abstract

For distributions $\mathbb{P}$ and $\mathbb{Q}$ with different supports or undefined densities, the divergence $\textrm{D}(\mathbb{P}||\mathbb{Q})$ may not exist. We define a Spread Divergence $\tilde{\textrm{D}}(\mathbb{P}||\mathbb{Q})$ on modified $\mathbb{P}$ and $\mathbb{Q}$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a Spread Divergence to train implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).