Depth-aware Fusion Method based on Image and 4D Radar Spectrum for 3D Object Detection

TL;DR

This paper presents a novel depth-aware fusion approach combining 4D radar spectra and camera images for improved 3D object detection in autonomous driving, especially under adverse weather conditions.

Contribution

It introduces a fusion method that integrates radar and camera data using attention mechanisms and GAN-based depth image generation, enhancing detection accuracy and robustness.

Findings

Improved 3D detection accuracy in various weather conditions.

Effective fusion of radar spectra and camera images in BEV perspective.

GAN-based depth image generation enhances detection when depth sensors are unavailable.

Abstract

Safety and reliability are crucial for the public acceptance of autonomous driving. To ensure accurate and reliable environmental perception, intelligent vehicles must exhibit accuracy and robustness in various environments. Millimeter-wave radar, known for its high penetration capability, can operate effectively in adverse weather conditions such as rain, snow, and fog. Traditional 3D millimeter-wave radars can only provide range, Doppler, and azimuth information for objects. Although the recent emergence of 4D millimeter-wave radars has added elevation resolution, the radar point clouds remain sparse due to Constant False Alarm Rate (CFAR) operations. In contrast, cameras offer rich semantic details but are sensitive to lighting and weather conditions. Hence, this paper leverages these two highly complementary and cost-effective sensors, 4D millimeter-wave radar and camera. By integrating 4D radar spectra with depth-aware camera images and employing attention mechanisms, we fuse texture-rich images with depth-rich radar data in the Bird's Eye View (BEV) perspective, enhancing 3D object detection. Additionally, we propose using GAN-based networks to generate depth images from radar spectra in the absence of depth sensors, further improving detection accuracy.