Cross-Dataset Experimental Study of Radar-Camera Fusion in Bird's-Eye View

TL;DR

This study evaluates a novel radar-camera fusion network for bird's-eye view perception, demonstrating improved performance over single-sensor systems across two datasets, with insights into data requirements and transfer learning benefits.

Contribution

Introduces a flexible radar-camera fusion network and provides a comprehensive evaluation on two datasets, highlighting the impact of data size and transfer learning.

Findings

Fusion outperforms single-sensor baselines

Camera needs large diverse data for optimal performance

Radar benefits from high-performance radar sensors

Abstract

By exploiting complementary sensor information, radar and camera fusion systems have the potential to provide a highly robust and reliable perception system for advanced driver assistance systems and automated driving functions. Recent advances in camera-based object detection offer new radar-camera fusion possibilities with bird's eye view feature maps. In this work, we propose a novel and flexible fusion network and evaluate its performance on two datasets: nuScenes and View-of-Delft. Our experiments reveal that while the camera branch needs large and diverse training data, the radar branch benefits more from a high-performance radar. Using transfer learning, we improve the camera's performance on the smaller dataset. Our results further demonstrate that the radar-camera fusion approach significantly outperforms the camera-only and radar-only baselines.