Physics Informed Distillation for Diffusion Models

TL;DR

This paper introduces Physics Informed Distillation (PID), a novel method that leverages the connection between diffusion models and ODEs, using PINNs principles to efficiently distill diffusion models without synthetic data, achieving competitive performance.

Contribution

The paper proposes PID, a new distillation approach for diffusion models that incorporates physics-informed principles, reducing complexity and improving usability compared to existing methods.

Findings

PID achieves performance comparable to recent distillation methods.

PID demonstrates predictable hyperparameter trends.

PID eliminates the need for synthetic dataset generation during distillation.

Abstract

Diffusion models have recently emerged as a potent tool in generative modeling. However, their inherent iterative nature often results in sluggish image generation due to the requirement for multiple model evaluations. Recent progress has unveiled the intrinsic link between diffusion models and Probability Flow Ordinary Differential Equations (ODEs), thus enabling us to conceptualize diffusion models as ODE systems. Simultaneously, Physics Informed Neural Networks (PINNs) have substantiated their effectiveness in solving intricate differential equations through implicit modeling of their solutions. Building upon these foundational insights, we introduce Physics Informed Distillation (PID), which employs a student model to represent the solution of the ODE system corresponding to the teacher diffusion model, akin to the principles employed in PINNs. Through experiments on CIFAR 10 and ImageNet 64x64, we observe that PID achieves performance comparable to recent distillation methods. Notably, it demonstrates predictable trends concerning method-specific hyperparameters and eliminates the need for synthetic dataset generation during the distillation process. Both of which contribute to its easy-to-use nature as a distillation approach for Diffusion Models. Our code and pre-trained checkpoint are publicly available at: https://github.com/pantheon5100/pid_diffusion.git.