Sparse Depth Completion with Semantic Mesh Deformation Optimization

TL;DR

This paper introduces a neural network with post-optimization for depth completion from sparse samples and RGB images, significantly improving accuracy over existing methods for indoor and outdoor scenes.

Contribution

It presents EDNet architecture, a semantic edge-weighted loss, and a semantic mesh deformation optimization, advancing depth completion techniques.

Findings

Reduces mean average error by up to 19.5% on NYU-Depth-V2

Outperforms existing methods on indoor and outdoor datasets

Improves depth prediction accuracy with novel optimization techniques

Abstract

Sparse depth measurements are widely available in many applications such as augmented reality, visual inertial odometry and robots equipped with low cost depth sensors. Although such sparse depth samples work well for certain applications like motion tracking, a complete depth map is usually preferred for broader applications, such as 3D object recognition, 3D reconstruction and autonomous driving. Despite the recent advancements in depth prediction from single RGB images with deeper neural networks, the existing approaches do not yield reliable results for practical use. In this work, we propose a neural network with post-optimization, which takes an RGB image and sparse depth samples as input and predicts the complete depth map. We make three major contributions to advance the state-of-the-art: an improved backbone network architecture named EDNet, a semantic edge-weighted loss function and a semantic mesh deformation optimization method. Our evaluation results outperform the existing work consistently on both indoor and outdoor datasets, and it significantly reduces the mean average error by up to 19.5% under the same settings of 200 sparse samples on NYU-Depth-V2 dataset.