Learning to Generate Novel Scene Compositions from Single Images and Videos

TL;DR

This paper introduces One-Shot GAN, a novel model capable of generating diverse and high-quality scene compositions from just one image or video, addressing low-data training challenges.

Contribution

It proposes a two-branch discriminator architecture that enables learning from a single image or video, improving diversity and quality in scene synthesis.

Findings

Achieves higher diversity and quality than previous single-image GANs.

Successfully learns from a single video, extending the single-image setting.

Generates plausible, novel scene compositions with preserved context.

Abstract

Training GANs in low-data regimes remains a challenge, as overfitting often leads to memorization or training divergence. In this work, we introduce One-Shot GAN that can learn to generate samples from a training set as little as one image or one video. We propose a two-branch discriminator, with content and layout branches designed to judge the internal content separately from the scene layout realism. This allows synthesis of visually plausible, novel compositions of a scene, with varying content and layout, while preserving the context of the original sample. Compared to previous single-image GAN models, One-Shot GAN achieves higher diversity and quality of synthesis. It is also not restricted to the single image setting, successfully learning in the introduced setting of a single video.