Self-supervised Sparse to Dense Motion Segmentation

TL;DR

This paper introduces a self-supervised approach for converting sparse motion segmentations into dense pixel-level segmentations in videos, operating on single frames without pre-training.

Contribution

It presents a novel self-supervised method that densifies sparse motion cues into dense segmentations without relying on large pre-training datasets.

Findings

Achieves high-quality dense motion segmentation from sparse input

Operates effectively on single frames without pre-training

Validated on FBMS59 and DAVIS16 datasets

Abstract

Observable motion in videos can give rise to the definition of objects moving with respect to the scene. The task of segmenting such moving objects is referred to as motion segmentation and is usually tackled either by aggregating motion information in long, sparse point trajectories, or by directly producing per frame dense segmentations relying on large amounts of training data. In this paper, we propose a self supervised method to learn the densification of sparse motion segmentations from single video frames. While previous approaches towards motion segmentation build upon pre-training on large surrogate datasets and use dense motion information as an essential cue for the pixelwise segmentation, our model does not require pre-training and operates at test time on single frames. It can be trained in a sequence specific way to produce high quality dense segmentations from sparse and noisy input. We evaluate our method on the well-known motion segmentation datasets FBMS59 and DAVIS16.