Stochastic Adversarial Video Prediction

TL;DR

This paper introduces a combined stochastic adversarial model for video prediction that produces more realistic and diverse future frames by integrating latent variable modeling with adversarial training, outperforming previous methods.

Contribution

It presents a novel approach that combines latent variational models with adversarial training to improve the realism and diversity of video predictions.

Findings

Predictions are more realistic to human raters.

The method covers a wider range of possible futures.

Outperforms prior models in realism and diversity.

Abstract

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.