MSDPN: Monocular Depth Prediction with Partial Laser Observation using Multi-stage Neural Networks

TL;DR

This paper introduces MSDPN, a multi-stage neural network that predicts dense depth maps from monocular images and 2D LiDAR data, effectively addressing partial observation issues with novel architecture and feature aggregation.

Contribution

The paper presents a new multi-stage encoder-decoder network with CSFA for improved depth prediction using real 2D LiDAR data, validated on a novel dataset.

Findings

MSDPN outperforms state-of-the-art methods in depth prediction accuracy.

The proposed architecture effectively mitigates partial observation problems.

Reference depth maps are robust even in untrained scenarios.

Abstract

In this study, a deep-learning-based multi-stage network architecture called Multi-Stage Depth Prediction Network (MSDPN) is proposed to predict a dense depth map using a 2D LiDAR and a monocular camera. Our proposed network consists of a multi-stage encoder-decoder architecture and Cross Stage Feature Aggregation (CSFA). The proposed multi-stage encoder-decoder architecture alleviates the partial observation problem caused by the characteristics of a 2D LiDAR, and CSFA prevents the multi-stage network from diluting the features and allows the network to learn the inter-spatial relationship between features better. Previous works use sub-sampled data from the ground truth as an input rather than actual 2D LiDAR data. In contrast, our approach trains the model and conducts experiments with a physically-collected 2D LiDAR dataset. To this end, we acquired our own dataset called KAIST RGBD-scan dataset and validated the effectiveness and the robustness of MSDPN under realistic conditions. As verified experimentally, our network yields promising performance against state-of-the-art methods. Additionally, we analyzed the performance of different input methods and confirmed that the reference depth map is robust in untrained scenarios.