NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation

TL;DR

NVDS+ is a versatile, efficient neural stabilizer that improves video depth estimation consistency across various models and applications, supported by a large-scale dataset and bidirectional inference strategy.

Contribution

Introduces NVDS+, a plug-and-play stabilizer for video depth estimation, along with the largest natural-scene video depth dataset VDW, and extends to multiple downstream tasks.

Findings

Significant improvements in depth consistency and accuracy

Effective across various single-image models and applications

Provides a large-scale dataset for training and evaluation

Abstract

Video depth estimation aims to infer temporally consistent depth. One approach is to finetune a single-image model on each video with geometry constraints, which proves inefficient and lacks robustness. An alternative is learning to enforce consistency from data, which requires well-designed models and sufficient video depth data. To address both challenges, we introduce NVDS+ that stabilizes inconsistent depth estimated by various single-image models in a plug-and-play manner. We also elaborate a large-scale Video Depth in the Wild (VDW) dataset, which contains 14,203 videos with over two million frames, making it the largest natural-scene video depth dataset. Additionally, a bidirectional inference strategy is designed to improve consistency by adaptively fusing forward and backward predictions. We instantiate a model family ranging from small to large scales for different applications. The method is evaluated on VDW dataset and three public benchmarks. To further prove the versatility, we extend NVDS+ to video semantic segmentation and several downstream applications like bokeh rendering, novel view synthesis, and 3D reconstruction. Experimental results show that our method achieves significant improvements in consistency, accuracy, and efficiency. Our work serves as a solid baseline and data foundation for learning-based video depth estimation. Code and dataset are available at: https://github.com/RaymondWang987/NVDS