Predicting Scene Parsing and Motion Dynamics in the Future

TL;DR

This paper introduces a novel joint model for predicting future scene parsing and optical flow in videos, enhancing understanding of scene dynamics for autonomous systems.

Contribution

It is the first model to jointly predict scene parsing and motion dynamics, leveraging their mutual benefits for improved accuracy.

Findings

Significantly better parsing and motion prediction results on Cityscapes dataset.

Effective joint modeling of scene semantics and motion improves future frame understanding.

Model can predict vehicle steering angles, demonstrating understanding of scene dynamics.

Abstract

The ability of predicting the future is important for intelligent systems, e.g. autonomous vehicles and robots to plan early and make decisions accordingly. Future scene parsing and optical flow estimation are two key tasks that help agents better understand their environments as the former provides dense semantic information, i.e. what objects will be present and where they will appear, while the latter provides dense motion information, i.e. how the objects will move. In this paper, we propose a novel model to simultaneously predict scene parsing and optical flow in unobserved future video frames. To our best knowledge, this is the first attempt in jointly predicting scene parsing and motion dynamics. In particular, scene parsing enables structured motion prediction by decomposing optical flow into different groups while optical flow estimation brings reliable pixel-wise correspondence to scene parsing. By exploiting this mutually beneficial relationship, our model shows significantly better parsing and motion prediction results when compared to well-established baselines and individual prediction models on the large-scale Cityscapes dataset. In addition, we also demonstrate that our model can be used to predict the steering angle of the vehicles, which further verifies the ability of our model to learn latent representations of scene dynamics.