Autoencoding Video Latents for Adversarial Video Generation

TL;DR

This paper introduces AVLAE, a novel adversarial autoencoder framework for disentangling motion and appearance in videos, improving robustness and control in video generation without relying on handcrafted architectures.

Contribution

The paper proposes AVLAE, a two-stream latent autoencoder that learns to disentangle motion and appearance in videos through adversarial training, bypassing the need for explicit structural priors.

Findings

Successfully disentangles motion and appearance without explicit priors

Achieves improved diversity and robustness in video generation

Demonstrates effectiveness through qualitative and quantitative evaluations

Abstract

Given the three dimensional complexity of a video signal, training a robust and diverse GAN based video generative model is onerous due to large stochasticity involved in data space. Learning disentangled representations of the data help to improve robustness and provide control in the sampling process. For video generation, there is a recent progress in this area by considering motion and appearance as orthogonal information and designing architectures that efficiently disentangle them. These approaches rely on handcrafting architectures that impose structural priors on the generator to decompose appearance and motion codes in the latent space. Inspired from the recent advancements in the autoencoder based image generation, we present AVLAE (Adversarial Video Latent AutoEncoder) which is a two stream latent autoencoder where the video distribution is learned by adversarial training. In particular, we propose to autoencode the motion and appearance latent vectors of the video generator in the adversarial setting. We demonstrate that our approach learns to disentangle motion and appearance codes even without the explicit structural composition in the generator. Several experiments with qualitative and quantitative results demonstrate the effectiveness of our method.