Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models

TL;DR

Patch Diffusion introduces a patch-wise training framework for diffusion models that significantly reduces training time and data requirements while maintaining or improving generation quality, enabling broader accessibility.

Contribution

The paper presents a novel patch-wise training method with a new conditional score function, improving efficiency and performance of diffusion models on small datasets.

Findings

Achieves ≥2× faster training times.

Maintains comparable or better image quality.

Excels on small datasets with high FID scores.

Abstract

Diffusion models are powerful, but they require a lot of time and data to train. We propose Patch Diffusion, a generic patch-wise training framework, to significantly reduce the training time costs while improving data efficiency, which thus helps democratize diffusion model training to broader users. At the core of our innovations is a new conditional score function at the patch level, where the patch location in the original image is included as additional coordinate channels, while the patch size is randomized and diversified throughout training to encode the cross-region dependency at multiple scales. Sampling with our method is as easy as in the original diffusion model. Through Patch Diffusion, we could achieve $\mathbf{\ge 2\times}$ faster training, while maintaining comparable or better generation quality. Patch Diffusion meanwhile improves the performance of diffusion models trained on relatively small datasets, $e.g.$, as few as 5,000 images to train from scratch. We achieve outstanding FID scores in line with state-of-the-art benchmarks: 1.77 on CelebA-64$\times$64, 1.93 on AFHQv2-Wild-64$\times$64, and 2.72 on ImageNet-256$\times$256. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Patch-Diffusion.