Conditional Distribution Modelling for Few-Shot Image Synthesis with Diffusion Models

TL;DR

This paper introduces Conditional Distribution Modelling (CDM), a novel framework leveraging diffusion models to improve diversity and fidelity in few-shot image synthesis by better approximating unseen class distributions.

Contribution

The paper proposes CDM, a new method that models latent space distributions to enhance diversity and fidelity in few-shot image generation using diffusion models.

Findings

Outperforms existing methods on four benchmark datasets

Improves diversity of generated images

Enhances fidelity to unseen classes

Abstract

Few-shot image synthesis entails generating diverse and realistic images of novel categories using only a few example images. While multiple recent efforts in this direction have achieved impressive results, the existing approaches are dependent only upon the few novel samples available at test time in order to generate new images, which restricts the diversity of the generated images. To overcome this limitation, we propose Conditional Distribution Modelling (CDM) -- a framework which effectively utilizes Diffusion models for few-shot image generation. By modelling the distribution of the latent space used to condition a Diffusion process, CDM leverages the learnt statistics of the training data to get a better approximation of the unseen class distribution, thereby removing the bias arising due to limited number of few shot samples. Simultaneously, we devise a novel inversion based optimization strategy that further improves the approximated unseen class distribution, and ensures the fidelity of the generated samples to the unseen class. The experimental results on four benchmark datasets demonstrate the effectiveness of our proposed CDM for few-shot generation.