Generating Novel Scene Compositions from Single Images and Videos

TL;DR

This paper introduces SIV-GAN, a novel generative model capable of creating diverse and high-quality scene compositions from a single image or video clip, addressing challenges of low-data regimes.

Contribution

The paper proposes a two-branch discriminator architecture for GANs that enables synthesis of diverse, high-quality scene compositions from minimal data, including single images and videos.

Findings

SIV-GAN produces more diverse images than previous models.

The model maintains high quality and realism in generated scenes.

It successfully learns from highly similar video frames.

Abstract

Given a large dataset for training, generative adversarial networks (GANs) can achieve remarkable performance for the image synthesis task. However, training GANs in extremely low data regimes remains a challenge, as overfitting often occurs, leading to memorization or training divergence. In this work, we introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or a single video clip. We propose a two-branch discriminator architecture, with content and layout branches designed to judge internal content and scene layout realism separately from each other. This discriminator design enables synthesis of visually plausible, novel compositions of a scene, with varying content and layout, while preserving the context of the original sample. Compared to previous single image GANs, our model generates more diverse, higher quality images, while not being restricted to a single image setting. We further introduce a new challenging task of learning from a few frames of a single video. In this training setup the training images are highly similar to each other, which makes it difficult for prior GAN models to achieve a synthesis of both high quality and diversity.