GANs Can Play Lottery Tickets Too

TL;DR

This paper investigates the existence of sparse, trainable subnetworks within GANs, demonstrating their potential for effective compression and transferability, and outperforming previous methods in image generation and translation tasks.

Contribution

First study of lottery ticket subnetworks in GANs, showing their existence at high sparsity and their transferability, leading to improved GAN compression methods.

Findings

Matching subnetworks exist at 67%-74% sparsity in GANs.

Discriminator initialization significantly affects subnetwork quality.

Subnetworks outperform state-of-the-art GAN compression techniques.

Abstract

Deep generative adversarial networks (GANs) have gained growing popularity in numerous scenarios, while usually suffer from high parameter complexities for resource-constrained real-world applications. However, the compression of GANs has less been explored. A few works show that heuristically applying compression techniques normally leads to unsatisfactory results, due to the notorious training instability of GANs. In parallel, the lottery ticket hypothesis shows prevailing success on discriminative models, in locating sparse matching subnetworks capable of training in isolation to full model performance. In this work, we for the first time study the existence of such trainable matching subnetworks in deep GANs. For a range of GANs, we certainly find matching subnetworks at 67%-74% sparsity. We observe that with or without pruning discriminator has a minor effect on the existence and quality of matching subnetworks, while the initialization weights used in the discriminator play a significant role. We then show the powerful transferability of these subnetworks to unseen tasks. Furthermore, extensive experimental results demonstrate that our found subnetworks substantially outperform previous state-of-the-art GAN compression approaches in both image generation (e.g. SNGAN) and image-to-image translation GANs (e.g. CycleGAN). Codes available at https://github.com/VITA-Group/GAN-LTH.