On the Anomalous Generalization of GANs

TL;DR

This paper investigates why GANs sometimes fail to learn the true data distribution, identifying two key problems—sample insufficiency and pixel-wise combination—that cause anomalous generalization, and proposes mitigation methods that improve performance.

Contribution

It uncovers two specific causes of anomalous GAN behavior and introduces methods to address these issues, enhancing GAN training stability and output quality.

Findings

Theoretical and empirical evidence of sample insufficiency affecting discriminator accuracy.

Discriminator can be fooled by pixel-wise combinations, leading to biased generator supervision.

Proposed mitigation methods improve FID scores by up to 30% on natural images.

Abstract

Generative models, especially Generative Adversarial Networks (GANs), have received significant attention recently. However, it has been observed that in terms of some attributes, e.g. the number of simple geometric primitives in an image, GANs are not able to learn the target distribution in practice. Motivated by this observation, we discover two specific problems of GANs leading to anomalous generalization behaviour, which we refer to as the sample insufficiency and the pixel-wise combination. For the first problem of sample insufficiency, we show theoretically and empirically that the batchsize of the training samples in practice may be insufficient for the discriminator to learn an accurate discrimination function. It could result in unstable training dynamics for the generator, leading to anomalous generalization. For the second problem of pixel-wise combination, we find that besides recognizing the positive training samples as real, under certain circumstances, the discriminator could be fooled to recognize the pixel-wise combinations (e.g. pixel-wise average) of the positive training samples as real. However, those combinations could be visually different from the real samples in the target distribution. With the fooled discriminator as reference, the generator would obtain biased supervision further, leading to the anomalous generalization behaviour. Additionally, in this paper, we propose methods to mitigate the anomalous generalization of GANs. Extensive experiments on benchmark show our proposed methods improve the FID score up to 30\% on natural image dataset.