FewGAN: Generating from the Joint Distribution of a Few Images

TL;DR

FewGAN is a hierarchical generative model that creates diverse, high-quality images from a small set of training samples by combining quantization, patch embeddings, and residual GANs.

Contribution

It introduces a novel hierarchical patch-GAN framework that leverages quantization and autoregressive modeling to generate images from limited data.

Findings

Outperforms baseline models quantitatively.

Produces high-quality, diverse images.

Effective with small training sets.

Abstract

We introduce FewGAN, a generative model for generating novel, high-quality and diverse images whose patch distribution lies in the joint patch distribution of a small number of N>1 training samples. The method is, in essence, a hierarchical patch-GAN that applies quantization at the first coarse scale, in a similar fashion to VQ-GAN, followed by a pyramid of residual fully convolutional GANs at finer scales. Our key idea is to first use quantization to learn a fixed set of patch embeddings for training images. We then use a separate set of side images to model the structure of generated images using an autoregressive model trained on the learned patch embeddings of training images. Using quantization at the coarsest scale allows the model to generate both conditional and unconditional novel images. Subsequently, a patch-GAN renders the fine details, resulting in high-quality images. In an extensive set of experiments, it is shown that FewGAN outperforms baselines both quantitatively and qualitatively.