RGB-Depth Fusion GAN for Indoor Depth Completion

TL;DR

This paper introduces a novel two-branch fusion network utilizing GANs and adaptive modules to improve indoor depth completion from RGB and incomplete depth images, especially in large missing regions.

Contribution

It proposes a new end-to-end fusion network with a GAN-based branch and adaptive fusion modules for enhanced depth completion in indoor scenes.

Findings

Significant improvement on NYU-Depth V2 and SUN RGB-D datasets.

Effective handling of large missing depth regions.

Enhanced depth map quality with the proposed fusion approach.

Abstract

The raw depth image captured by the indoor depth sensor usually has an extensive range of missing depth values due to inherent limitations such as the inability to perceive transparent objects and limited distance range. The incomplete depth map burdens many downstream vision tasks, and a rising number of depth completion methods have been proposed to alleviate this issue. While most existing methods can generate accurate dense depth maps from sparse and uniformly sampled depth maps, they are not suitable for complementing the large contiguous regions of missing depth values, which is common and critical. In this paper, we design a novel two-branch end-to-end fusion network, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. The first branch employs an encoder-decoder structure to regress the local dense depth values from the raw depth map, with the help of local guidance information extracted from the RGB image. In the other branch, we propose an RGB-depth fusion GAN to transfer the RGB image to the fine-grained textured depth map. We adopt adaptive fusion modules named W-AdaIN to propagate the features across the two branches, and we append a confidence fusion head to fuse the two outputs of the branches for the final depth map. Extensive experiments on NYU-Depth V2 and SUN RGB-D demonstrate that our proposed method clearly improves the depth completion performance, especially in a more realistic setting of indoor environments with the help of the pseudo depth map.