Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth

TL;DR

This paper introduces a locally weighted linear regression approach to recover scale and shift in monocular video depth estimation, improving consistency and accuracy in 3D scene reconstruction.

Contribution

It proposes a novel method for scale and shift recovery in monocular video depth estimation, enhancing existing approaches and enabling robust 3D scene reconstruction.

Findings

Boosts state-of-the-art performance by up to 50% on benchmarks.

Trains a strong depth model surpassing DPT ViT-Large.

Enables accurate 3D scene shape recovery from monocular videos.

Abstract

Existing monocular depth estimation methods have achieved excellent robustness in diverse scenes, but they can only retrieve affine-invariant depth, up to an unknown scale and shift. However, in some video-based scenarios such as video depth estimation and 3D scene reconstruction from a video, the unknown scale and shift residing in per-frame prediction may cause the depth inconsistency. To solve this problem, we propose a locally weighted linear regression method to recover the scale and shift with very sparse anchor points, which ensures the scale consistency along consecutive frames. Extensive experiments show that our method can boost the performance of existing state-of-the-art approaches by 50% at most over several zero-shot benchmarks. Besides, we merge over 6.3 million RGBD images to train strong and robust depth models. Our produced ResNet50-backbone model even outperforms the state-of-the-art DPT ViT-Large model. Combining with geometry-based reconstruction methods, we formulate a new dense 3D scene reconstruction pipeline, which benefits from both the scale consistency of sparse points and the robustness of monocular methods. By performing the simple per-frame prediction over a video, the accurate 3D scene shape can be recovered.