Stochastic Video Generation with a Learned Prior

TL;DR

This paper introduces an unsupervised stochastic video generation model that learns a prior of uncertainty, producing varied and sharp future frames that outperform existing methods.

Contribution

It presents a novel learned prior approach for stochastic video generation that captures uncertainty and improves the quality of long-term predictions.

Findings

Generated videos are more varied and sharper than previous methods.

The model is simple, end-to-end trainable, and effective across datasets.

Sample generations remain high quality even many frames into the future.

Abstract

Generating video frames that accurately predict future world states is challenging. Existing approaches either fail to capture the full distribution of outcomes, or yield blurry generations, or both. In this paper we introduce an unsupervised video generation model that learns a prior model of uncertainty in a given environment. Video frames are generated by drawing samples from this prior and combining them with a deterministic estimate of the future frame. The approach is simple and easily trained end-to-end on a variety of datasets. Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.