Diverse Generation from a Single Video Made Possible

TL;DR

This paper introduces a non-parametric, classical approach for diverse video generation and manipulation from a single video, outperforming GANs in quality, speed, and scalability, and enabling practical applications.

Contribution

The paper demonstrates that classical space-time patches-nearest-neighbors methods can surpass single-video GANs in quality and efficiency, providing a scalable, learning-free baseline.

Findings

Outperforms single-video GANs in visual quality and realism

Reduces runtime from days to seconds

Enables practical diverse video generation and manipulation

Abstract

GANs are able to perform generation and manipulation tasks, trained on a single video. However, these single video GANs require unreasonable amount of time to train on a single video, rendering them almost impractical. In this paper we question the necessity of a GAN for generation from a single video, and introduce a non-parametric baseline for a variety of generation and manipulation tasks. We revive classical space-time patches-nearest-neighbors approaches and adapt them to a scalable unconditional generative model, without any learning. This simple baseline surprisingly outperforms single-video GANs in visual quality and realism (confirmed by quantitative and qualitative evaluations), and is disproportionately faster (runtime reduced from several days to seconds). Other than diverse video generation, we demonstrate other applications using the same framework, including video analogies and spatio-temporal retargeting. Our proposed approach is easily scaled to Full-HD videos. These observations show that the classical approaches, if adapted correctly, significantly outperform heavy deep learning machinery for these tasks. This sets a new baseline for single-video generation and manipulation tasks, and no less important -- makes diverse generation from a single video practically possible for the first time.