Recurrent Deconvolutional Generative Adversarial Networks with Application to Text Guided Video Generation

TL;DR

This paper introduces RD-GAN, a recurrent deconvolutional GAN that effectively generates realistic videos conditioned on text descriptions, addressing frame discontinuity and leveraging long-range temporal dependencies.

Contribution

It presents a novel recurrent deconvolutional generator and a 3D CNN discriminator for improved text-guided video synthesis and related tasks.

Findings

Successfully generates realistic videos conditioned on text

Achieves superior performance in video prediction and classification

Addresses frame discontinuity issues in video generation

Abstract

This paper proposes a novel model for video generation and especially makes the attempt to deal with the problem of video generation from text descriptions, i.e., synthesizing realistic videos conditioned on given texts. Existing video generation methods cannot be easily adapted to handle this task well, due to the frame discontinuity issue and their text-free generation schemes. To address these problems, we propose a recurrent deconvolutional generative adversarial network (RD-GAN), which includes a recurrent deconvolutional network (RDN) as the generator and a 3D convolutional neural network (3D-CNN) as the discriminator. The RDN is a deconvolutional version of conventional recurrent neural network, which can well model the long-range temporal dependency of generated video frames and make good use of conditional information. The proposed model can be jointly trained by pushing the RDN to generate realistic videos so that the 3D-CNN cannot distinguish them from real ones. We apply the proposed RD-GAN to a series of tasks including conventional video generation, conditional video generation, video prediction and video classification, and demonstrate its effectiveness by achieving well performance.