PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher

TL;DR

PaGoDA introduces a progressive training pipeline for diffusion autoencoders that significantly reduces training costs and achieves state-of-the-art image generation quality across multiple resolutions.

Contribution

It presents a novel three-stage pipeline for diffusion models that combines downsampled training, distillation, and super-resolution, enabling efficient high-quality image synthesis.

Findings

Achieves 64x reduction in training cost on 8x downsampled data.

State-of-the-art performance on ImageNet across multiple resolutions.

Applicable in latent space for compression and autoencoder integration.

Abstract

The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose Progressive Growing of Diffusion Autoencoder (PaGoDA), a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64\times$ reduced cost in training its diffusion model on 8x downsampled data; while at the inference, with the single-step, it performs state-of-the-art on ImageNet across all resolutions from 64x64 to 512x512, and text-to-image. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g., Stable Diffusion). The code is available at https://github.com/sony/pagoda.