RadarRGBD A Multi-Sensor Fusion Dataset for Perception with RGB-D and mmWave Radar

TL;DR

RadarRGBD is a comprehensive multi-sensor dataset combining RGB-D, high-resolution millimeter-wave radar, and raw radar data, designed to advance perception research in challenging environments like low-light and adverse weather conditions.

Contribution

The paper introduces RadarRGBD, a new dataset with high-resolution radar and raw data, and proposes a depth estimation method that improves depth map quality using dataset information.

Findings

Effective filling of missing regions in depth maps

Enhanced depth estimation accuracy in challenging conditions

Dataset supports multi-sensor fusion research

Abstract

Multi-sensor fusion has significant potential in perception tasks for both indoor and outdoor environments. Especially under challenging conditions such as adverse weather and low-light environments, the combined use of millimeter-wave radar and RGB-D sensors has shown distinct advantages. However, existing multi-sensor datasets in the fields of autonomous driving and robotics often lack high-quality millimeter-wave radar data. To address this gap, we present a new multi-sensor dataset:RadarRGBD. This dataset includes RGB-D data, millimeter-wave radar point clouds, and raw radar matrices, covering various indoor and outdoor scenes, as well as low-light environments. Compared to existing datasets, RadarRGBD employs higher-resolution millimeter-wave radar and provides raw data, offering a new research foundation for the fusion of millimeter-wave radar and visual sensors. Furthermore, to tackle the noise and gaps in depth maps captured by Kinect V2 due to occlusions and mismatches, we fine-tune an open-source relative depth estimation framework, incorporating the absolute depth information from the dataset for depth supervision. We also introduce pseudo-relative depth scale information to further optimize the global depth scale estimation. Experimental results demonstrate that the proposed method effectively fills in missing regions in sensor data. Our dataset and related documentation will be publicly available at: https://github.com/song4399/RadarRGBD.