Consistency Guided Scene Flow Estimation

TL;DR

This paper introduces a self-supervised framework called Consistency Guided Scene Flow Estimation (CGSF) that jointly reconstructs 3D scene structure and motion from stereo video, with iterative refinement and domain adaptation capabilities.

Contribution

The paper proposes a novel self-supervised method with a learned refinement network and iterative test-time adaptation for improved scene flow estimation.

Findings

Achieves better generalization than state-of-the-art methods.

Reliably predicts disparity and scene flow in challenging scenarios.

Adapts quickly and robustly to unseen domains.

Abstract

Consistency Guided Scene Flow Estimation (CGSF) is a self-supervised framework for the joint reconstruction of 3D scene structure and motion from stereo video. The model takes two temporal stereo pairs as input, and predicts disparity and scene flow. The model self-adapts at test time by iteratively refining its predictions. The refinement process is guided by a consistency loss, which combines stereo and temporal photo-consistency with a geometric term that couples disparity and 3D motion. To handle inherent modeling error in the consistency loss (e.g. Lambertian assumptions) and for better generalization, we further introduce a learned, output refinement network, which takes the initial predictions, the loss, and the gradient as input, and efficiently predicts a correlated output update. In multiple experiments, including ablation studies, we show that the proposed model can reliably predict disparity and scene flow in challenging imagery, achieves better generalization than the state-of-the-art, and adapts quickly and robustly to unseen domains.