A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving

TL;DR

This survey reviews deep learning-based radar-vision fusion techniques for 3D object detection in autonomous driving, highlighting recent methods, classifications, and future trends in the field.

Contribution

It provides a comprehensive overview and classification of radar-vision fusion strategies, including recent advancements like 4D radar applications, for autonomous vehicle perception.

Findings

Deep learning enhances radar-vision fusion performance.

End-to-end fusion methods are currently most promising.

4D radar offers new capabilities in autonomous driving.

Abstract

With the rapid advancement of autonomous driving technology, there is a growing need for enhanced safety and efficiency in the automatic environmental perception of vehicles during their operation. In modern vehicle setups, cameras and mmWave radar (radar), being the most extensively employed sensors, demonstrate complementary characteristics, inherently rendering them conducive to fusion and facilitating the achievement of both robust performance and cost-effectiveness. This paper focuses on a comprehensive survey of radar-vision (RV) fusion based on deep learning methods for 3D object detection in autonomous driving. We offer a comprehensive overview of each RV fusion category, specifically those employing region of interest (ROI) fusion and end-to-end fusion strategies. As the most promising fusion strategy at present, we provide a deeper classification of end-to-end fusion methods, including those 3D bounding box prediction based and BEV based approaches. Moreover, aligning with recent advancements, we delineate the latest information on 4D radar and its cutting-edge applications in autonomous vehicles (AVs). Finally, we present the possible future trends of RV fusion and summarize this paper.