Video Generation from Text Employing Latent Path Construction for Temporal Modeling

TL;DR

This paper introduces a novel method for generating realistic videos from natural language descriptions by regressing latent representations and employing a progressive upsampling approach, outperforming existing baselines on complex datasets.

Contribution

It presents the first approach for text-to-video generation on realistic datasets, using latent path construction and a stacking upPooling block for progressive frame synthesis.

Findings

Outperforms RNN and deconvolution-based methods

Effective in generating videos from complex natural language descriptions

Capable of handling realistic datasets like A2D and UCF101

Abstract

Video generation is one of the most challenging tasks in Machine Learning and Computer Vision fields of study. In this paper, we tackle the text to video generation problem, which is a conditional form of video generation. Humans can listen/read natural language sentences, and can imagine or visualize what is being described; therefore, we believe that video generation from natural language sentences will have an important impact on Artificial Intelligence. Video generation is relatively a new field of study in Computer Vision, which is far from being solved. The majority of recent works deal with synthetic datasets or real datasets with very limited types of objects, scenes, and emotions. To the best of our knowledge, this is the very first work on the text (free-form sentences) to video generation on more realistic video datasets like Actor and Action Dataset (A2D) or UCF101. We tackle the complicated problem of video generation by regressing the latent representations of the first and last frames and employing a context-aware interpolation method to build the latent representations of in-between frames. We propose a stacking ``upPooling'' block to sequentially generate RGB frames out of each latent representation and progressively increase the resolution. Moreover, our proposed Discriminator encodes videos based on single and multiple frames. We provide quantitative and qualitative results to support our arguments and show the superiority of our method over well-known baselines like Recurrent Neural Network (RNN) and Deconvolution (as known as Convolutional Transpose) based video generation methods.