Transfer Learning for Diffusion Models

TL;DR

This paper introduces TGDP, a novel transfer learning method for diffusion models that combines pre-trained source models with domain guidance, improving performance with limited target data.

Contribution

The paper proposes TGDP, a new transfer learning approach for diffusion models that integrates source models with domain classifiers and extends to conditional modeling.

Findings

TGDP effectively transfers knowledge to target domains with limited data.

The method outperforms traditional finetuning and regularization approaches.

TGDP demonstrates strong results on both simulated and real-world datasets.

Abstract

Diffusion models, a specific type of generative model, have achieved unprecedented performance in recent years and consistently produce high-quality synthetic samples. A critical prerequisite for their notable success lies in the presence of a substantial number of training samples, which can be impractical in real-world applications due to high collection costs or associated risks. Consequently, various finetuning and regularization approaches have been proposed to transfer knowledge from existing pre-trained models to specific target domains with limited data. This paper introduces the Transfer Guided Diffusion Process (TGDP), a novel approach distinct from conventional finetuning and regularization methods. We prove that the optimal diffusion model for the target domain integrates pre-trained diffusion models on the source domain with additional guidance from a domain classifier. We further extend TGDP to a conditional version for modeling the joint distribution of data and its corresponding labels, together with two additional regularization terms to enhance the model performance. We validate the effectiveness of TGDP on both simulated and real-world datasets.