R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

TL;DR

R4Det introduces a novel fusion framework combining radar and camera data to improve 3D object detection in autonomous driving, addressing depth estimation, temporal fusion, and small object detection challenges.

Contribution

It proposes new modules for depth fusion, pose-independent temporal fusion, and semantic refinement, achieving state-of-the-art results in 3D detection.

Findings

Achieves state-of-the-art results on TJ4DRadSet and VoD datasets.

Enhances depth estimation through Panoramic Depth Fusion.

Develops a pose-independent temporal fusion module.

Abstract

4D radar-camera sensing configuration has gained increasing importance in autonomous driving. However, existing 3D object detection methods that fuse 4D Radar and camera data confront several challenges. First, their absolute depth estimation module is not robust and accurate enough, leading to inaccurate 3D localization. Second, the performance of their temporal fusion module will degrade dramatically or even fail when the ego vehicle's pose is missing or inaccurate. Third, for some small objects, the sparse radar point clouds may completely fail to reflect from their surfaces. In such cases, detection must rely solely on visual unimodal priors. To address these limitations, we propose R4Det, which enhances depth estimation quality via the Panoramic Depth Fusion module, enabling mutual reinforcement between absolute and relative depth. For temporal fusion, we design a Deformable Gated Temporal Fusion module that does not rely on the ego vehicle's pose. In addition, we built an Instance-Guided Dynamic Refinement module that extracts semantic prototypes from 2D instance guidance. Experiments show that R4Det achieves state-of-the-art 3D object detection results on the TJ4DRadSet and VoD datasets. The source code and models will be released at https://github.com/VDIGPKU/R4Det.