RCM-Fusion: Radar-Camera Multi-Level Fusion for 3D Object Detection

TL;DR

This paper introduces RCM-Fusion, a novel radar-camera multi-level fusion approach that enhances 3D object detection by effectively combining features and instances, achieving state-of-the-art results on the nuScenes dataset.

Contribution

The paper presents a new multi-level fusion framework that integrates radar and camera data at feature and instance levels, improving 3D detection accuracy.

Findings

Achieves state-of-the-art performance on nuScenes benchmark.

Effectively fuses radar and camera data at multiple levels.

Reduces localization errors with the Radar Grid Point Refinement module.

Abstract

While LiDAR sensors have been successfully applied to 3D object detection, the affordability of radar and camera sensors has led to a growing interest in fusing radars and cameras for 3D object detection. However, previous radar-camera fusion models were unable to fully utilize the potential of radar information. In this paper, we propose Radar-Camera Multi-level fusion (RCM-Fusion), which attempts to fuse both modalities at both feature and instance levels. For feature-level fusion, we propose a Radar Guided BEV Encoder which transforms camera features into precise BEV representations using the guidance of radar Bird's-Eye-View (BEV) features and combines the radar and camera BEV features. For instance-level fusion, we propose a Radar Grid Point Refinement module that reduces localization error by accounting for the characteristics of the radar point clouds. The experiments conducted on the public nuScenes dataset demonstrate that our proposed RCM-Fusion achieves state-of-the-art performances among single frame-based radar-camera fusion methods in the nuScenes 3D object detection benchmark. Code will be made publicly available.