Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed

TL;DR

This paper introduces a knowledge distillation technique that condenses multi-step denoising processes into a single step, significantly accelerating iterative generative models while maintaining high sample quality.

Contribution

It presents a novel distillation method that transforms slow iterative models into fast single-step generators without adversarial training.

Findings

Denoising Student achieves comparable quality to GANs on CIFAR-10 and CelebA.

The method scales effectively to high-resolution images like 256x256 LSUN.

Sampling speed improves by 2-3 orders of magnitude.

Abstract

Iterative generative models, such as noise conditional score networks and denoising diffusion probabilistic models, produce high quality samples by gradually denoising an initial noise vector. However, their denoising process has many steps, making them 2-3 orders of magnitude slower than other generative models such as GANs and VAEs. In this paper, we establish a novel connection between knowledge distillation and image generation with a technique that distills a multi-step denoising process into a single step, resulting in a sampling speed similar to other single-step generative models. Our Denoising Student generates high quality samples comparable to GANs on the CIFAR-10 and CelebA datasets, without adversarial training. We demonstrate that our method scales to higher resolutions through experiments on 256 x 256 LSUN. Code and checkpoints are available at https://github.com/tcl9876/Denoising_Student