Fast Multi-frame Stereo Scene Flow with Motion Segmentation

TL;DR

This paper introduces a fast, unified multi-frame approach for scene flow, depth, optical flow, and motion segmentation from stereo videos, achieving high accuracy and efficiency, especially on challenging sequences.

Contribution

The method combines stereo matching, visual odometry, and motion segmentation into a unified framework, significantly improving speed and accuracy over prior approaches.

Findings

Ranks third on KITTI scene flow benchmark.

Runs in 2-3 seconds per frame on CPU, much faster than previous methods.

Outperforms existing methods on challenging Sintel sequences.

Abstract

We propose a new multi-frame method for efficiently computing scene flow (dense depth and optical flow) and camera ego-motion for a dynamic scene observed from a moving stereo camera rig. Our technique also segments out moving objects from the rigid scene. In our method, we first estimate the disparity map and the 6-DOF camera motion using stereo matching and visual odometry. We then identify regions inconsistent with the estimated camera motion and compute per-pixel optical flow only at these regions. This flow proposal is fused with the camera motion-based flow proposal using fusion moves to obtain the final optical flow and motion segmentation. This unified framework benefits all four tasks - stereo, optical flow, visual odometry and motion segmentation leading to overall higher accuracy and efficiency. Our method is currently ranked third on the KITTI 2015 scene flow benchmark. Furthermore, our CPU implementation runs in 2-3 seconds per frame which is 1-3 orders of magnitude faster than the top six methods. We also report a thorough evaluation on challenging Sintel sequences with fast camera and object motion, where our method consistently outperforms OSF [Menze and Geiger, 2015], which is currently ranked second on the KITTI benchmark.