$\alpha$-GAN by R\'{e}nyi Cross Entropy

TL;DR

This paper introduces $\alpha$-GAN, a novel generative adversarial network leveraging R\'enyi measures, which enhances convergence speed and potentially addresses issues like vanishing gradients by adjusting the R\'enyi order.

Contribution

The paper formulates a new $\alpha$-GAN framework based on R\'enyi cross entropy, generalizing vanilla GAN and exploring the effects of different R\'enyi orders on training dynamics.

Findings

Gradient amplification for $\alpha \\in (0,1)$ accelerates convergence.

Choosing $\alpha \\in (0,1)$ may mitigate vanishing gradient problems.

Experimental results confirm faster training with $\alpha$ in the range (0,1).

Abstract

This paper proposes $\alpha$-GAN, a generative adversarial network using R\'{e}nyi measures. The value function is formulated, by R\'{e}nyi cross entropy, as an expected certainty measure incurred by the discriminator's soft decision as to where the sample is from, true population or the generator. The discriminator tries to maximize the R\'{e}nyi certainty about sample source, while the generator wants to reduce it by injecting fake samples. This forms a min-max problem with the solution parameterized by the R\'{e}nyi order $\alpha$. This $\alpha$-GAN reduces to vanilla GAN at $\alpha = 1$, where the value function is exactly the binary cross entropy. The optimization of $\alpha$-GAN is over probability (vector) space. It is shown that the gradient is exponentially enlarged when R\'{e}nyi order is in the range $\alpha \in (0,1)$. This makes convergence faster, which is verified by experimental results. A discussion shows that choosing $\alpha \in (0,1)$ may be able to solve some common problems, e.g., vanishing gradient. A following observation reveals that this range has not been fully explored in the existing R\'{e}nyi version GANs.