Zero-Shot Monocular Scene Flow Estimation in the Wild

TL;DR

This paper introduces a zero-shot monocular scene flow estimation method that generalizes well across diverse real-world scenes, overcoming data scarcity and parameterization challenges to improve practical applicability.

Contribution

It proposes a joint geometry-motion estimation approach, a large-scale synthetic training data recipe, and an effective scene flow parameterization, enabling zero-shot generalization in the wild.

Findings

Outperforms existing scene flow methods in 3D end-point error

Demonstrates zero-shot generalization to casual videos and robotic scenes

Creates 1 million annotated synthetic training samples

Abstract

Large models have shown generalization across datasets for many low-level vision tasks, like depth estimation, but no such general models exist for scene flow. Even though scene flow has wide potential use, it is not used in practice because current predictive models do not generalize well. We identify three key challenges and propose solutions for each. First, we create a method that jointly estimates geometry and motion for accurate prediction. Second, we alleviate scene flow data scarcity with a data recipe that affords us 1M annotated training samples across diverse synthetic scenes. Third, we evaluate different parameterizations for scene flow prediction and adopt a natural and effective parameterization. Our resulting model outperforms existing methods as well as baselines built on large-scale models in terms of 3D end-point error, and shows zero-shot generalization to the casually captured videos from DAVIS and the robotic manipulation scenes from RoboTAP. Overall, our approach makes scene flow prediction more practical in-the-wild.