Cross-Level Sensor Fusion with Object Lists via Transformer for 3D Object Detection

TL;DR

This paper introduces an end-to-end Transformer-based method for 3D object detection that fuses processed object lists with raw camera images, improving accuracy and training efficiency in automotive sensor systems.

Contribution

It presents the first cross-level fusion approach combining object lists and raw images using Transformer, along with a novel pseudo-label generation method for training.

Findings

Significant performance gains over baseline on nuScenes dataset

Effective generalization across various noise levels in object lists

Accelerated training convergence due to Gaussian mask integration

Abstract

In automotive sensor fusion systems, smart sensors and Vehicle-to-Everything (V2X) modules are commonly utilized. Sensor data from these systems are typically available only as processed object lists rather than raw sensor data from traditional sensors. Instead of processing other raw data separately and then fusing them at the object level, we propose an end-to-end cross-level fusion concept with Transformer, which integrates highly abstract object list information with raw camera images for 3D object detection. Object lists are fed into a Transformer as denoising queries and propagated together with learnable queries through the latter feature aggregation process. Additionally, a deformable Gaussian mask, derived from the positional and size dimensional priors from the object lists, is explicitly integrated into the Transformer decoder. This directs attention toward the target area of interest and accelerates model training convergence. Furthermore, as there is no public dataset containing object lists as a standalone modality, we propose an approach to generate pseudo object lists from ground-truth bounding boxes by simulating state noise and false positives and negatives. As the first work to conduct cross-level fusion, our approach shows substantial performance improvements over the vision-based baseline on the nuScenes dataset. It demonstrates its generalization capability over diverse noise levels of simulated object lists and real detectors.