Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure

TL;DR

This paper introduces an unsupervised method for realistic video generation from a single frame by estimating 3D scene structure and motion, enabling coherent future frame prediction without labeled data.

Contribution

It proposes a novel end-to-end model that estimates 3D scene structure and predicts future frames solely from a single image, without requiring 3D or segmentation annotations.

Findings

Successfully estimates 3D structure and motion segmentation from one frame

Generates plausible and varied future video predictions

Works effectively on challenging natural video datasets

Abstract

Our goal in this work is to generate realistic videos given just one initial frame as input. Existing unsupervised approaches to this task do not consider the fact that a video typically shows a 3D environment, and that this should remain coherent from frame to frame even as the camera and objects move. We address this by developing a model that first estimates the latent 3D structure of the scene, including the segmentation of any moving objects. It then predicts future frames by simulating the object and camera dynamics, and rendering the resulting views. Importantly, it is trained end-to-end using only the unsupervised objective of predicting future frames, without any 3D information nor segmentation annotations. Experiments on two challenging datasets of natural videos show that our model can estimate 3D structure and motion segmentation from a single frame, and hence generate plausible and varied predictions.