Generalization of GANs and overparameterized models under Lipschitz continuity

TL;DR

This paper introduces a Lipschitz-based theoretical framework to explain and improve the generalization of GANs and overparameterized neural networks, revealing why Lipschitz constraints and normalization techniques enhance performance.

Contribution

It develops a new Lipschitz theory for analyzing generalization, derives bounds for GANs, and explains the empirical success of Lipschitz constraints and normalization methods.

Findings

Lipschitz penalization improves GAN generalization.

Dropout and spectral normalization enable deep networks to generalize well.

The theory explains empirical success of Lipschitz constraints in GANs.

Abstract

Generative adversarial networks (GANs) are so complex that the existing learning theories do not provide a satisfactory explanation for why GANs have great success in practice. The same situation also remains largely open for deep neural networks. To fill this gap, we introduce a Lipschitz theory to analyze generalization. We demonstrate its simplicity by analyzing generalization and consistency of overparameterized neural networks. We then use this theory to derive Lipschitz-based generalization bounds for GANs. Our bounds show that penalizing the Lipschitz constant of the GAN loss can improve generalization. This result answers the long mystery of why the popular use of Lipschitz constraint for GANs often leads to great success, empirically without a solid theory. Finally but surprisingly, we show that, when using Dropout or spectral normalization, both \emph{truly deep} neural networks and GANs can generalize well without the curse of dimensionality.