Stochastic Video Prediction with Structure and Motion

TL;DR

This paper introduces a method for stochastic video prediction that disentangles static scene structure from dynamic motion, improving prediction accuracy in complex driving scenarios by modeling foreground and background separately.

Contribution

It proposes a novel approach to factorize video observations into static and dynamic components, enhancing the modeling of complex scene dynamics in stochastic video prediction.

Findings

Disentangling structure and motion improves prediction quality.

The method outperforms existing models on KITTI and Cityscapes datasets.

Separate modeling of foreground and background enhances scene understanding.

Abstract

While stochastic video prediction models enable future prediction under uncertainty, they mostly fail to model the complex dynamics of real-world scenes. For example, they cannot provide reliable predictions for scenes with a moving camera and independently moving foreground objects in driving scenarios. The existing methods fail to fully capture the dynamics of the structured world by only focusing on changes in pixels. In this paper, we assume that there is an underlying process creating observations in a video and propose to factorize it into static and dynamic components. We model the static part based on the scene structure and the ego-motion of the vehicle, and the dynamic part based on the remaining motion of the dynamic objects. By learning separate distributions of changes in foreground and background, we can decompose the scene into static and dynamic parts and separately model the change in each. Our experiments demonstrate that disentangling structure and motion helps stochastic video prediction, leading to better future predictions in complex driving scenarios on two real-world driving datasets, KITTI and Cityscapes.