Propagating Sparse Depth via Depth Foundation Model for Out-of-Distribution Depth Completion

TL;DR

This paper introduces a novel depth completion framework that leverages depth foundation models and dual-space propagation to achieve robust depth reconstruction in out-of-distribution scenarios without extensive training.

Contribution

The proposed method uses depth foundation models and a dual-space propagation approach to enhance robustness in depth completion, especially for out-of-distribution data.

Findings

Outperforms state-of-the-art methods in OOD scenarios

Effective propagation of sparse depth in 3D and 2D spaces

No large-scale training required for robustness

Abstract

Depth completion is a pivotal challenge in computer vision, aiming at reconstructing the dense depth map from a sparse one, typically with a paired RGB image. Existing learning based models rely on carefully prepared but limited data, leading to significant performance degradation in out-of-distribution (OOD) scenarios. Recent foundation models have demonstrated exceptional robustness in monocular depth estimation through large-scale training, and using such models to enhance the robustness of depth completion models is a promising solution. In this work, we propose a novel depth completion framework that leverages depth foundation models to attain remarkable robustness without large-scale training. Specifically, we leverage a depth foundation model to extract environmental cues, including structural and semantic context, from RGB images to guide the propagation of sparse depth information into missing regions. We further design a dual-space propagation approach, without any learnable parameters, to effectively propagates sparse depth in both 3D and 2D spaces to maintain geometric structure and local consistency. To refine the intricate structure, we introduce a learnable correction module to progressively adjust the depth prediction towards the real depth. We train our model on the NYUv2 and KITTI datasets as in-distribution datasets and extensively evaluate the framework on 16 other datasets. Our framework performs remarkably well in the OOD scenarios and outperforms existing state-of-the-art depth completion methods. Our models are released in https://github.com/shenglunch/PSD.