Compressing GANs using Knowledge Distillation

TL;DR

This paper introduces a knowledge distillation approach to significantly compress GANs, enabling their deployment on low-resource devices while maintaining high-quality image generation.

Contribution

The authors propose a novel method to compress large GANs using knowledge distillation, achieving high compression ratios with preserved image quality.

Findings

Achieved compression ratios of 1669:1, 58:1, and 87:1 on MNIST, CIFAR-10, and Celeb-A datasets.

Compressed GANs outperform standard-trained GANs at the same parameter budget.

Identified a qualitative limit for GAN compression while retaining quality.

Abstract

Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardware, such as mobile devices, and for applications with real time capabilities. There has been no work found to reduce the number of parameters used in GANs. Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller "student" GAN learns to mimic a larger "teacher" GAN. We show that the distillation methods used on MNIST, CIFAR-10, and Celeb-A datasets can compress teacher GANs at ratios of 1669:1, 58:1, and 87:1, respectively, while retaining the quality of the generated image. From our experiments, we observe a qualitative limit for GAN's compression. Moreover, we observe that, with a fixed parameter budget, compressed GANs outperform GANs trained using standard training methods. We conjecture that this is partially owing to the optimization landscape of over-parameterized GANs which allows efficient training using alternating gradient descent. Thus, training an over-parameterized GAN followed by our proposed compression scheme provides a high quality generative model with a small number of parameters.