Spatio-Temporal Image Boundary Extrapolation

TL;DR

This paper introduces a novel approach for predicting future image boundaries in videos, demonstrating the ability to extrapolate motion and physics-based boundary trajectories without relying on object models.

Contribution

It presents the first method for spatio-temporal boundary extrapolation in videos, capable of long-term physics-based predictions without strong parametric assumptions.

Findings

Successful boundary prediction in real-world videos

Long-term physics-based boundary extrapolation demonstrated

No reliance on object-specific models or assumptions

Abstract

Boundary prediction in images as well as video has been a very active topic of research and organizing visual information into boundaries and segments is believed to be a corner stone of visual perception. While prior work has focused on predicting boundaries for observed frames, our work aims at predicting boundaries of future unobserved frames. This requires our model to learn about the fate of boundaries and extrapolate motion patterns. We experiment on established real-world video segmentation dataset, which provides a testbed for this new task. We show for the first time spatio-temporal boundary extrapolation in this challenging scenario. Furthermore, we show long-term prediction of boundaries in situations where the motion is governed by the laws of physics. We successfully predict boundaries in a billiard scenario without any assumptions of a strong parametric model or any object notion. We argue that our model has with minimalistic model assumptions derived a notion of 'intuitive physics' that can be applied to novel scenes.