Adversarial Video Generation on Complex Datasets

TL;DR

This paper introduces DVD-GAN, a scalable adversarial model that generates high-fidelity, complex videos on large datasets, advancing the state-of-the-art in video synthesis and prediction.

Contribution

The paper presents DVD-GAN, a novel scalable GAN architecture with an efficient discriminator decomposition for high-quality video generation on complex datasets.

Findings

Achieved new state-of-the-art Fréchet Inception Distance on Kinetics-600 prediction.

Achieved state-of-the-art Inception Score on UCF-101 for video synthesis.

Established a strong baseline for video synthesis on Kinetics-600.

Abstract

Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fr\'echet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.