Multi-Modal Depth Estimation Using Convolutional Neural Networks

TL;DR

This paper proposes a deep learning method for dense depth estimation by fusing Radar and camera data, demonstrating its effectiveness on multiple datasets including real-world and synthetic scenarios.

Contribution

It introduces a novel CNN-based depth estimation approach that leverages Radar and camera fusion with transfer learning, addressing challenges in adverse weather conditions.

Findings

Effective depth estimation on Nuscenes, KITTI, and synthetic datasets.

Successful application to safety-critical crane operation scenarios.

Outperforms existing methods in sensor fusion for depth prediction.

Abstract

This paper addresses the problem of dense depth predictions from sparse distance sensor data and a single camera image on challenging weather conditions. This work explores the significance of different sensor modalities such as camera, Radar, and Lidar for estimating depth by applying Deep Learning approaches. Although Lidar has higher depth-sensing abilities than Radar and has been integrated with camera images in lots of previous works, depth estimation using CNN's on the fusion of robust Radar distance data and camera images has not been explored much. In this work, a deep regression network is proposed utilizing a transfer learning approach consisting of an encoder where a high performing pre-trained model has been used to initialize it for extracting dense features and a decoder for upsampling and predicting desired depth. The results are demonstrated on Nuscenes, KITTI, and a Synthetic dataset which was created using the CARLA simulator. Also, top-view zoom-camera images captured from the crane on a construction site are evaluated to estimate the distance of the crane boom carrying heavy loads from the ground to show the usability in safety-critical applications.