GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks

TL;DR

GraN-GAN introduces a novel gradient normalization technique that enforces a piecewise Lipschitz constraint in GAN discriminators, leading to improved image generation performance across various datasets and metrics.

Contribution

The paper proposes Gradient Normalization (GraN), a new method that strictly enforces a piecewise Lipschitz constraint in GAN discriminators, differing from spectral normalization and gradient penalties.

Findings

Enhanced image generation quality on multiple datasets.

Significant performance improvements by tuning the Lipschitz constant K.

Insights into the relationship between K, training dynamics, and optimizer behavior.

Abstract

Modern generative adversarial networks (GANs) predominantly use piecewise linear activation functions in discriminators (or critics), including ReLU and LeakyReLU. Such models learn piecewise linear mappings, where each piece handles a subset of the input space, and the gradients per subset are piecewise constant. Under such a class of discriminator (or critic) functions, we present Gradient Normalization (GraN), a novel input-dependent normalization method, which guarantees a piecewise K-Lipschitz constraint in the input space. In contrast to spectral normalization, GraN does not constrain processing at the individual network layers, and, unlike gradient penalties, strictly enforces a piecewise Lipschitz constraint almost everywhere. Empirically, we demonstrate improved image generation performance across multiple datasets (incl. CIFAR-10/100, STL-10, LSUN bedrooms, and CelebA), GAN loss functions, and metrics. Further, we analyze altering the often untuned Lipschitz constant K in several standard GANs, not only attaining significant performance gains, but also finding connections between K and training dynamics, particularly in low-gradient loss plateaus, with the common Adam optimizer.