DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild

TL;DR

This paper introduces DATAP, a dynamic-aware point tracking method that improves camera trajectory estimation and dense point cloud reconstruction in challenging dynamic videos by leveraging consistent depth and global optimization.

Contribution

The paper presents a novel dynamic-aware tracking approach that enhances structure from motion in dynamic scenes by integrating consistent depth and global bundle adjustment, reducing errors from traditional methods.

Findings

Achieves state-of-the-art camera pose estimation in dynamic scenes

Effective in complex real-world videos like DAVIS

Improves robustness over traditional optical flow-based methods

Abstract

This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. Traditional frameworks, such as ParticleSfM~\cite{zhao2022particlesfm}, address this problem by sequentially computing the optical flow between adjacent frames to obtain point trajectories. They then remove dynamic trajectories through motion segmentation and perform global bundle adjustment. However, the process of estimating optical flow between two adjacent frames and chaining the matches can introduce cumulative errors. Additionally, motion segmentation combined with single-view depth estimation often faces challenges related to scale ambiguity. To tackle these challenges, we propose a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking. Specifically, our DATAP addresses these issues by estimating dense point tracking across the video sequence and predicting the visibility and dynamics of each point. By incorporating the consistent video depth prior, the performance of motion segmentation is enhanced. With the integration of DATAP, it becomes possible to estimate and optimize all camera poses simultaneously by performing global bundle adjustments for point tracking classified as static and visible, rather than relying on incremental camera registration. Extensive experiments on dynamic sequences, e.g., Sintel and TUM RGBD dynamic sequences, and on the wild video, e.g., DAVIS, demonstrate that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.