Diffusion Model Patching via Mixture-of-Prompts

TL;DR

Diffusion Model Patching (DMP) introduces a small, learnable prompt set with a gating mechanism to improve pre-trained diffusion models' performance without retraining from scratch.

Contribution

DMP is a novel method that dynamically combines prompts to enhance frozen diffusion models, requiring minimal additional parameters and training.

Findings

DMP improves FID of converged models by over 10%.

Only 1.43% parameter increase needed for significant performance gains.

Effective even with additional training on original datasets.

Abstract

We present Diffusion Model Patching (DMP), a simple method to boost the performance of pre-trained diffusion models that have already reached convergence, with a negligible increase in parameters. DMP inserts a small, learnable set of prompts into the model's input space while keeping the original model frozen. The effectiveness of DMP is not merely due to the addition of parameters but stems from its dynamic gating mechanism, which selects and combines a subset of learnable prompts at every timestep (i.e., reverse denoising steps). This strategy, which we term "mixture-of-prompts", enables the model to draw on the distinct expertise of each prompt, essentially "patching" the model's functionality at every timestep with minimal yet specialized parameters. Uniquely, DMP enhances the model by further training on the original dataset already used for pre-training, even in a scenario where significant improvements are typically not expected due to model convergence. Notably, DMP significantly enhances the FID of converged DiT-L/2 by 10.38% on FFHQ, achieved with only a 1.43% parameter increase and 50K additional training iterations.