Flow and Depth Assisted Video Prediction with Latent Transformer

TL;DR

This paper introduces a novel approach for occluded video prediction by integrating depth and point-flow information into a latent transformer model, significantly improving prediction accuracy in occlusion scenarios.

Contribution

The study systematically explores the use of depth and point-flow data to enhance video prediction models, addressing occlusion challenges in both synthetic and real-world datasets.

Findings

Improved prediction accuracy with depth and point-flow assistance.

Better modeling of background motion in occluded scenarios.

Enhanced performance measured by appearance metrics and Wasserstein distances.

Abstract

Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occlusion is still an inherent challenge in video prediction. We hypothesize that providing explicit information about motion (via point-flow) and geometric structure (via depth-maps) will enable video prediction models to perform better in situations with occlusion and the background motion. To investigate this, we present the first systematic study dedicated to occluded video prediction. We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow. We evaluate this model in a controlled setting on both synthetic and real-world datasets with not only appearance-based metrics but also Wasserstein distances on object masks, which can effectively measure the motion distribution of the prediction. We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.