Winning Lottery Tickets in Deep Generative Models

TL;DR

This paper demonstrates the existence of lottery tickets in deep generative models like GANs and VAEs, showing that sparse sub-networks can be found early in training, leading to resource-efficient training and better initializations.

Contribution

It extends the lottery ticket hypothesis to generative models, introduces effective pruning methods, and shows transferability and early detection of winning tickets in these models.

Findings

Winning tickets can be found in GANs, VAEs, and AutoEncoders with high sparsity.

Early-bird tickets enable significant reductions in FLOPs and training time.

Transferability of winning tickets across different generative models.

Abstract

The lottery ticket hypothesis suggests that sparse, sub-networks of a given neural network, if initialized properly, can be trained to reach comparable or even better performance to that of the original network. Prior works in lottery tickets have primarily focused on the supervised learning setup, with several papers proposing effective ways of finding "winning tickets" in classification problems. In this paper, we confirm the existence of winning tickets in deep generative models such as GANs and VAEs. We show that the popular iterative magnitude pruning approach (with late rewinding) can be used with generative losses to find the winning tickets. This approach effectively yields tickets with sparsity up to 99% for AutoEncoders, 93% for VAEs and 89% for GANs on CIFAR and Celeb-A datasets. We also demonstrate the transferability of winning tickets across different generative models (GANs and VAEs) sharing the same architecture, suggesting that winning tickets have inductive biases that could help train a wide range of deep generative models. Furthermore, we show the practical benefits of lottery tickets in generative models by detecting tickets at very early stages in training called "early-bird tickets". Through early-bird tickets, we can achieve up to 88% reduction in floating-point operations (FLOPs) and 54% reduction in training time, making it possible to train large-scale generative models over tight resource constraints. These results out-perform existing early pruning methods like SNIP (Lee, Ajanthan, and Torr 2019) and GraSP (Wang, Zhang, and Grosse 2020). Our findings shed light towards existence of proper network initializations that could improve convergence and stability of generative models.