Efficient Transfer Learning in Diffusion Models via Adversarial Noise

TL;DR

This paper introduces TAN, a transfer learning method for diffusion probabilistic models that enhances image generation quality and diversity in limited data scenarios through similarity-guided training and adversarial noise selection.

Contribution

It presents a novel transfer learning approach specifically designed for DPMs, addressing the data scarcity challenge with two innovative strategies.

Findings

Outperforms existing GAN-based and DDPM-based methods in few-shot image generation.

Improves image quality and diversity with limited training data.

Demonstrates efficiency and effectiveness in extensive experiments.

Abstract

Diffusion Probabilistic Models (DPMs) have demonstrated substantial promise in image generation tasks but heavily rely on the availability of large amounts of training data. Previous works, like GANs, have tackled the limited data problem by transferring pre-trained models learned with sufficient data. However, those methods are hard to be utilized in DPMs since the distinct differences between DPM-based and GAN-based methods, showing in the unique iterative denoising process integral and the need for many timesteps with no-targeted noise in DPMs. In this paper, we propose a novel DPMs-based transfer learning method, TAN, to address the limited data problem. It includes two strategies: similarity-guided training, which boosts transfer with a classifier, and adversarial noise selection which adaptive chooses targeted noise based on the input image. Extensive experiments in the context of few-shot image generation tasks demonstrate that our method is not only efficient but also excels in terms of image quality and diversity when compared to existing GAN-based and DDPM-based methods.