StereoDiff: Stereo-Diffusion Synergy for Video Depth Estimation

TL;DR

StereoDiff introduces a two-stage video depth estimation method that combines stereo matching for static regions and diffusion models for dynamic regions, achieving state-of-the-art results with improved consistency and accuracy.

Contribution

The paper proposes a novel synergy of stereo matching and diffusion models tailored for static and dynamic regions in videos, addressing limitations of existing image-based methods.

Findings

Achieves state-of-the-art performance on real-world video benchmarks.

Demonstrates superior temporal consistency in depth estimation.

Effectively handles both static backgrounds and dynamic objects.

Abstract

Recent video depth estimation methods achieve great performance by following the paradigm of image depth estimation, i.e., typically fine-tuning pre-trained video diffusion models with massive data. However, we argue that video depth estimation is not a naive extension of image depth estimation. The temporal consistency requirements for dynamic and static regions in videos are fundamentally different. Consistent video depth in static regions, typically backgrounds, can be more effectively achieved via stereo matching across all frames, which provides much stronger global 3D cues. While the consistency for dynamic regions still should be learned from large-scale video depth data to ensure smooth transitions, due to the violation of triangulation constraints. Based on these insights, we introduce StereoDiff, a two-stage video depth estimator that synergizes stereo matching for mainly the static areas with video depth diffusion for maintaining consistent depth transitions in dynamic areas. We mathematically demonstrate how stereo matching and video depth diffusion offer complementary strengths through frequency domain analysis, highlighting the effectiveness of their synergy in capturing the advantages of both. Experimental results on zero-shot, real-world, dynamic video depth benchmarks, both indoor and outdoor, demonstrate StereoDiff's SoTA performance, showcasing its superior consistency and accuracy in video depth estimation.