Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles

TL;DR

This paper introduces a novel radar-camera fusion framework for autonomous vehicles that improves object detection accuracy and distance estimation by combining radar and image data through a middle-fusion approach.

Contribution

The paper presents a new radar-camera fusion architecture with a middle-fusion strategy and a Radar Proposal Refinement network for enhanced object detection and distance estimation.

Findings

Outperforms existing radar-camera fusion methods on nuScenes dataset

Achieves higher accuracy in 2D object detection

Provides precise distance estimations for objects

Abstract

In this paper we present a novel radar-camera sensor fusion framework for accurate object detection and distance estimation in autonomous driving scenarios. The proposed architecture uses a middle-fusion approach to fuse the radar point clouds and RGB images. Our radar object proposal network uses radar point clouds to generate 3D proposals from a set of 3D prior boxes. These proposals are mapped to the image and fed into a Radar Proposal Refinement (RPR) network for objectness score prediction and box refinement. The RPR network utilizes both radar information and image feature maps to generate accurate object proposals and distance estimations. The radar-based proposals are combined with image-based proposals generated by a modified Region Proposal Network (RPN). The RPN has a distance regression layer for estimating distance for every generated proposal. The radar-based and image-based proposals are merged and used in the next stage for object classification. Experiments on the challenging nuScenes dataset show our method outperforms other existing radar-camera fusion methods in the 2D object detection task while at the same time accurately estimates objects' distances.