Center Feature Fusion: Selective Multi-Sensor Fusion of Center-based Objects

TL;DR

This paper introduces Center Feature Fusion, a novel multi-sensor fusion method that selectively combines camera and LiDAR data at object centers, significantly improving 3D detection accuracy with fewer features.

Contribution

The paper proposes a new center-based fusion approach that reduces computational load and enhances detection performance by focusing on relevant object locations.

Findings

Outperforms LiDAR-only baseline by 4.9% mAP on nuScenes

Fuses up to 100x fewer features than existing methods

Effective in leveraging center-based detection for multi-sensor fusion

Abstract

Leveraging multi-modal fusion, especially between camera and LiDAR, has become essential for building accurate and robust 3D object detection systems for autonomous vehicles. Until recently, point decorating approaches, in which point clouds are augmented with camera features, have been the dominant approach in the field. However, these approaches fail to utilize the higher resolution images from cameras. Recent works projecting camera features to the bird's-eye-view (BEV) space for fusion have also been proposed, however they require projecting millions of pixels, most of which only contain background information. In this work, we propose a novel approach Center Feature Fusion (CFF), in which we leverage center-based detection networks in both the camera and LiDAR streams to identify relevant object locations. We then use the center-based detection to identify the locations of pixel features relevant to object locations, a small fraction of the total number in the image. These are then projected and fused in the BEV frame. On the nuScenes dataset, we outperform the LiDAR-only baseline by 4.9% mAP while fusing up to 100x fewer features than other fusion methods.