Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion

TL;DR

This paper introduces Sparse Auxiliary Networks (SANs), a novel architecture that enables monocular depth prediction and completion from RGB images with optional sparse depth measurements, achieving state-of-the-art results.

Contribution

We propose SANs, a new module that allows a single network to perform both depth prediction and completion depending on available data, using sparse convolutions and feature injection.

Findings

SANs achieve state-of-the-art depth prediction accuracy.

The architecture effectively handles both tasks simultaneously.

Experimental results on NYUv2, KITTI, and DDAD benchmarks validate its superiority.

Abstract

Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image (monodepth) with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Auxiliary Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion, depending on whether only RGB images or also sparse point clouds are available at inference time. First, we decouple the image and depth map encoding stages using sparse convolutions to process only the valid depth map pixels. Second, we inject this information, when available, into the skip connections of the depth prediction network, augmenting its features. Through extensive experimental analysis on one indoor (NYUv2) and two outdoor (KITTI and DDAD) benchmarks, we demonstrate that our proposed SAN architecture is able to simultaneously learn both tasks, while achieving a new state of the art in depth prediction by a significant margin.