Why Spectral Normalization Stabilizes GANs: Analysis and Improvements

TL;DR

This paper explains how spectral normalization stabilizes GAN training by controlling gradient issues, connects it to initialization methods, and proposes an improved normalization technique that enhances stability and sample quality.

Contribution

The paper provides a theoretical analysis of spectral normalization's effectiveness, links it to initialization strategies, and introduces Bidirectional Scaled Spectral Normalization (BSSN) for improved performance.

Findings

SN controls exploding and vanishing gradients in GANs

BSSN outperforms SN in stability and sample quality

Theoretical analysis links SN to LeCun initialization

Abstract

Spectral normalization (SN) is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, there is currently limited understanding of why SN is effective. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization. This connection helps to explain why the most popular implementation of SN for GANs requires no hyper-parameter tuning, whereas stricter implementations of SN have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Building on this theoretical understanding, we propose a new spectral normalization technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorporates insights from later improvements to LeCun initialization: Xavier initialization and Kaiming initialization. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.