Decoder Modulation for Indoor Depth Completion

TL;DR

This paper introduces a novel decoder modulation approach using SPADE blocks for indoor depth completion, effectively handling semi-dense sensor data and improving state-of-the-art results on multiple datasets.

Contribution

A new decoder modulation branch with SPADE blocks and a training strategy for semi-dense data, advancing indoor depth completion methods.

Findings

Achieves state-of-the-art results on Matterport3D dataset.

Performs competitively with LiDAR-based methods on KITTI.

Training strategy improves predictions without dense ground truth.

Abstract

Depth completion recovers a dense depth map from sensor measurements. Current methods are mostly tailored for very sparse depth measurements from LiDARs in outdoor settings, while for indoor scenes Time-of-Flight (ToF) or structured light sensors are mostly used. These sensors provide semi-dense maps, with dense measurements in some regions and almost empty in others. We propose a new model that takes into account the statistical difference between such regions. Our main contribution is a new decoder modulation branch added to the encoder-decoder architecture. The encoder extracts features from the concatenated RGB image and raw depth. Given the mask of missing values as input, the proposed modulation branch controls the decoding of a dense depth map from these features differently for different regions. This is implemented by modifying the spatial distribution of output signals inside the decoder via Spatially-Adaptive Denormalization (SPADE) blocks. Our second contribution is a novel training strategy that allows us to train on a semi-dense sensor data when the ground truth depth map is not available. Our model achieves the state of the art results on indoor Matterport3D dataset. Being designed for semi-dense input depth, our model is still competitive with LiDAR-oriented approaches on the KITTI dataset. Our training strategy significantly improves prediction quality with no dense ground truth available, as validated on the NYUv2 dataset.