Few-Shot Diffusion Models

TL;DR

Few-Shot Diffusion Models (FSDM) enable high-quality image generation from new classes with only a few samples by leveraging conditional DDPMs and a set-based Vision Transformer for effective few-shot learning.

Contribution

The paper introduces FSDM, a novel framework that adapts diffusion models for few-shot generation using patch-based set conditioning with a Vision Transformer.

Findings

FSDM can generate diverse images from unseen classes with as few as 5 samples.

Conditioning on patch-based set information improves training convergence.

FSDM outperforms baseline diffusion models in few-shot learning benchmarks.

Abstract

Denoising diffusion probabilistic models (DDPM) are powerful hierarchical latent variable models with remarkable sample generation quality and training stability. These properties can be attributed to parameter sharing in the generative hierarchy, as well as a parameter-free diffusion-based inference procedure. In this paper, we present Few-Shot Diffusion Models (FSDM), a framework for few-shot generation leveraging conditional DDPMs. FSDMs are trained to adapt the generative process conditioned on a small set of images from a given class by aggregating image patch information using a set-based Vision Transformer (ViT). At test time, the model is able to generate samples from previously unseen classes conditioned on as few as 5 samples from that class. We empirically show that FSDM can perform few-shot generation and transfer to new datasets. We benchmark variants of our method on complex vision datasets for few-shot learning and compare to unconditional and conditional DDPM baselines. Additionally, we show how conditioning the model on patch-based input set information improves training convergence.