Multiframe Scene Flow with Piecewise Rigid Motion

TL;DR

This paper presents a novel multiframe scene flow method that jointly optimizes appearance consistency and local rigid motions from RGB-D sequences, achieving high accuracy and robustness without prior motion assumptions.

Contribution

It introduces a global non-linear least squares formulation with oversegmentation and robust optimization, enabling real-time capable, accurate scene flow estimation for various motion types.

Findings

Outperforms state-of-the-art methods on synthetic and real data.

Handles rigid, piecewise rigid, articulated, and moderate non-rigid motions.

Achieves a new level of accuracy in RGB-D scene flow estimation.

Abstract

We introduce a novel multiframe scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences. In contrast to the competing methods, we take advantage of an oversegmentation of the reference frame and robust optimization techniques. We formulate scene flow recovery as a global non-linear least squares problem which is iteratively solved by a damped Gauss-Newton approach. As a result, we obtain a qualitatively new level of accuracy in RGB-D based scene flow estimation which can potentially run in real-time. Our method can handle challenging cases with rigid, piecewise rigid, articulated and moderate non-rigid motion, and does not rely on prior knowledge about the types of motions and deformations. Extensive experiments on synthetic and real data show that our method outperforms state-of-the-art.