Inference-based GAN Video Generation

TL;DR

This paper introduces a novel memory-efficient VAE-GAN-based framework for generating long, coherent videos with hundreds or thousands of frames, overcoming previous limitations in temporal scaling and quality degradation.

Contribution

It extends existing VAE-GAN models by employing a Markov chain approach with a recall mechanism to produce long, continuous videos with maintained temporal dependencies.

Findings

Successfully generates long videos with hundreds of frames.

Maintains temporal continuity and realistic movement in generated videos.

Outperforms previous models in long sequence generation quality.

Abstract

Video generation has seen remarkable progress thanks to advancements in generative deep learning. However, generating long sequences remains a significant challenge. Generated videos should not only display coherent and continuous movement but also meaningful movement in successions of scenes. Models such as GANs, VAEs, and Diffusion Networks have been used for generating short video sequences, typically up to 16 frames. In this paper, we first propose a new type of video generator by enabling adversarial-based unconditional video generators with a variational encoder, akin to a VAE-GAN hybrid structure. The proposed model, as in other video deep learning-based processing frameworks, incorporates two processing branches, one for content and another for movement. However, existing models struggle with the temporal scaling of the generated videos. Classical approaches often result in degraded video quality when attempting to increase the generated video length, especially for significantly long sequences. To overcome this limitation, our research study extends the initially proposed VAE-GAN video generation model by employing a novel, memory-efficient approach to generate long videos composed of hundreds or thousands of frames ensuring their temporal continuity, consistency and dynamics. Our approach leverages a Markov chain framework with a recall mechanism, where each state represents a short-length VAE-GAN video generator. This setup enables the sequential connection of generated video sub-sequences, maintaining temporal dependencies and resulting in meaningful long video sequences.