Panoptic Segmentation Forecasting

TL;DR

This paper introduces panoptic segmentation forecasting, a new approach for predicting future scene states by decomposing scenes into 'things' and 'stuff', enabling better autonomous agent decision-making.

Contribution

It proposes a novel task and model for forecasting scene decomposition into foreground objects and background, improving over existing methods.

Findings

The model outperforms baselines on the new leaderboard.

Decomposition into 'things' and 'stuff' enhances forecasting accuracy.

Forecasting background odometry and object dynamics improves scene prediction.

Abstract

Our goal is to forecast the near future given a set of recent observations. We think this ability to forecast, i.e., to anticipate, is integral for the success of autonomous agents which need not only passively analyze an observation but also must react to it in real-time. Importantly, accurate forecasting hinges upon the chosen scene decomposition. We think that superior forecasting can be achieved by decomposing a dynamic scene into individual 'things' and background 'stuff'. Background 'stuff' largely moves because of camera motion, while foreground 'things' move because of both camera and individual object motion. Following this decomposition, we introduce panoptic segmentation forecasting. Panoptic segmentation forecasting opens up a middle-ground between existing extremes, which either forecast instance trajectories or predict the appearance of future image frames. To address this task we develop a two-component model: one component learns the dynamics of the background stuff by anticipating odometry, the other one anticipates the dynamics of detected things. We establish a leaderboard for this novel task, and validate a state-of-the-art model that outperforms available baselines.