VAE^2: Preventing Posterior Collapse of Variational Video Predictions in the Wild

TL;DR

This paper introduces VAE$^2$, a novel VAE architecture designed to prevent posterior collapse in video prediction tasks with deterministic data, enabling more diverse and accurate future frame predictions.

Contribution

The paper proposes VAE$^2$, which explicitly incorporates stochasticity into the VAE to mitigate posterior collapse in deterministic video datasets, improving diversity of predictions.

Findings

VAE$^2$ effectively reduces posterior collapse in video prediction.

VAE$^2$ produces more diverse future frame predictions.

VAE$^2$ outperforms existing VAE-based methods on Cityscapes dataset.

Abstract

Predicting future frames of video sequences is challenging due to the complex and stochastic nature of the problem. Video prediction methods based on variational auto-encoders (VAEs) have been a great success, but they require the training data to contain multiple possible futures for an observed video sequence. This is hard to be fulfilled when videos are captured in the wild where any given observation only has a determinate future. As a result, training a vanilla VAE model with these videos inevitably causes posterior collapse. To alleviate this problem, we propose a novel VAE structure, dabbed VAE-in-VAE or VAE$^2$. The key idea is to explicitly introduce stochasticity into the VAE. We treat part of the observed video sequence as a random transition state that bridges its past and future, and maximize the likelihood of a Markov Chain over the video sequence under all possible transition states. A tractable lower bound is proposed for this intractable objective function and an end-to-end optimization algorithm is designed accordingly. VAE$^2$ can mitigate the posterior collapse problem to a large extent, as it breaks the direct dependence between future and observation and does not directly regress the determinate future provided by the training data. We carry out experiments on a large-scale dataset called Cityscapes, which contains videos collected from a number of urban cities. Results show that VAE$^2$ is capable of predicting diverse futures and is more resistant to posterior collapse than the other state-of-the-art VAE-based approaches. We believe that VAE$^2$ is also applicable to other stochastic sequence prediction problems where training data are lack of stochasticity.