LCF3D: A Robust and Real-Time Late-Cascade Fusion Framework for 3D Object Detection in Autonomous Driving

TL;DR

LCF3D is a real-time sensor fusion framework that improves 3D object detection in autonomous driving by combining RGB and LiDAR data through late and cascade fusion techniques, enhancing detection accuracy and domain robustness.

Contribution

It introduces a novel late-cascade fusion framework that effectively combines 2D RGB detections with 3D LiDAR detections, reducing false positives and recovering missed objects.

Findings

Significant improvement over LiDAR-only methods on KITTI and nuScenes datasets.

Enhanced detection of pedestrians, cyclists, motorcycles, and bicycles.

Robustness to different sensor configurations across domains.

Abstract

Accurately localizing 3D objects like pedestrians, cyclists, and other vehicles is essential in Autonomous Driving. To ensure high detection performance, Autonomous Vehicles complement RGB cameras with LiDAR sensors, but effectively combining these data sources for 3D object detection remains challenging. We propose LCF3D, a novel sensor fusion framework that combines a 2D object detector on RGB images with a 3D object detector on LiDAR point clouds. By leveraging multimodal fusion principles, we compensate for inaccuracies in the LiDAR object detection network. Our solution combines two key principles: (i) late fusion, to reduce LiDAR False Positives by matching LiDAR 3D detections with RGB 2D detections and filtering out unmatched LiDAR detections; and (ii) cascade fusion, to recover missed objects from LiDAR by generating new 3D frustum proposals corresponding to unmatched RGB detections. Experiments show that LCF3D is beneficial for domain generalization, as it turns out to be successful in handling different sensor configurations between training and testing domains. LCF3D achieves significant improvements over LiDAR-based methods, particularly for challenging categories like pedestrians and cyclists in the KITTI dataset, as well as motorcycles and bicycles in nuScenes. Code can be downloaded from: https://github.com/CarloSgaravatti/LCF3D.