Temporally Consistent Motion Segmentation from RGB-D Video

TL;DR

This paper introduces a method for consistent motion segmentation in RGB-D videos, enabling the reconstruction of individual 3D objects by segmenting and tracking their motion over entire sequences.

Contribution

It proposes a novel energy formulation and initialization technique for temporally consistent segmentation and motion estimation in RGB-D sequences.

Findings

Enables fusion of object segments into 3D reconstructions.

Uses a coordinate descent approach for energy minimization.

Leverages depth information for improved clustering.

Abstract

We present a method for temporally consistent motion segmentation from RGB-D videos assuming a piecewise rigid motion model. We formulate global energies over entire RGB-D sequences in terms of the segmentation of each frame into a number of objects, and the rigid motion of each object through the sequence. We develop a novel initialization procedure that clusters feature tracks obtained from the RGB data by leveraging the depth information. We minimize the energy using a coordinate descent approach that includes novel techniques to assemble object motion hypotheses. A main benefit of our approach is that it enables us to fuse consistently labeled object segments from all RGB-D frames of an input sequence into individual 3D object reconstructions.