RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion

TL;DR

This paper introduces RigNet++, a novel depth completion network that employs repetitive image guidance and semantic assistance to improve the accuracy of dense depth map recovery from sparse data, achieving state-of-the-art results.

Contribution

The paper proposes a repetitive design in both image guidance and depth generation branches, integrating semantic priors from SAM and a new dataset TOFDC for enhanced depth completion.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively models high-frequency structures with reduced complexity.

Utilizes semantic prior to refine depth maps significantly.

Abstract

Depth completion aims to recover dense depth maps from sparse ones, where color images are often used to facilitate this task. Recent depth methods primarily focus on image guided learning frameworks. However, blurry guidance in the image and unclear structure in the depth still impede their performance. To tackle these challenges, we explore a repetitive design in our image guided network to gradually and sufficiently recover depth values. Specifically, the repetition is embodied in both the image guidance branch and depth generation branch. In the former branch, we design a dense repetitive hourglass network (DRHN) to extract discriminative image features of complex environments, which can provide powerful contextual instruction for depth prediction. In the latter branch, we present a repetitive guidance (RG) module based on dynamic convolution, in which an efficient convolution factorization is proposed to reduce the complexity while modeling high-frequency structures progressively. Furthermore, in the semantic guidance branch, we utilize the well-known large vision model, i.e., segment anything (SAM), to supply RG with semantic prior. In addition, we propose a region-aware spatial propagation network (RASPN) for further depth refinement based on the semantic prior constraint. Finally, we collect a new dataset termed TOFDC for the depth completion task, which is acquired by the time-of-flight (TOF) sensor and the color camera on smartphones. Extensive experiments demonstrate that our method achieves state-of-the-art performance on KITTI, NYUv2, Matterport3D, 3D60, VKITTI, and our TOFDC.