RoMo: Robust Motion Segmentation Improves Structure from Motion

TL;DR

RoMo is a novel iterative method that combines optical flow, epipolar cues, and pre-trained segmentation to improve motion segmentation, thereby enhancing structure-from-motion and camera calibration in dynamic scenes.

Contribution

Introduces RoMo, a simple yet effective approach that significantly improves motion segmentation and camera calibration in dynamic scenes by integrating multiple cues with a pre-trained model.

Findings

Outperforms unsupervised and synthetic supervised baselines in motion segmentation.

Establishes new state-of-the-art in camera calibration for scenes with dynamic content.

Enhances SfM pipelines with robust motion segmentation.

Abstract

There has been extensive progress in the reconstruction and generation of 4D scenes from monocular casually-captured video. While these tasks rely heavily on known camera poses, the problem of finding such poses using structure-from-motion (SfM) often depends on robustly separating static from dynamic parts of a video. The lack of a robust solution to this problem limits the performance of SfM camera-calibration pipelines. We propose a novel approach to video-based motion segmentation to identify the components of a scene that are moving w.r.t. a fixed world frame. Our simple but effective iterative method, RoMo, combines optical flow and epipolar cues with a pre-trained video segmentation model. It outperforms unsupervised baselines for motion segmentation as well as supervised baselines trained from synthetic data. More importantly, the combination of an off-the-shelf SfM pipeline with our segmentation masks establishes a new state-of-the-art on camera calibration for scenes with dynamic content, outperforming existing methods by a substantial margin.