Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving

TL;DR

This paper introduces a graph-based multi-modal sensor fusion framework for autonomous driving, improving scene understanding and object tracking accuracy by integrating diverse sensor data through a novel online state estimation technique.

Contribution

It presents the first online graph-aware Kalman filter for fusing multi-modal sensor graphs, enhancing multi-object tracking and scene perception in autonomous systems.

Findings

Improved MOTA scores on real-world datasets

Reduced position errors and identity switches

Effective integration of heterogeneous sensor data

Abstract

The growing demand for robust scene understanding in mobile robotics and autonomous driving has highlighted the importance of integrating multiple sensing modalities. By combining data from diverse sensors like cameras and LIDARs, fusion techniques can overcome the limitations of individual sensors, enabling a more complete and accurate perception of the environment. We introduce a novel approach to multi-modal sensor fusion, focusing on developing a graph-based state representation that supports critical decision-making processes in autonomous driving. We present a Sensor-Agnostic Graph-Aware Kalman Filter [3], the first online state estimation technique designed to fuse multi-modal graphs derived from noisy multi-sensor data. The estimated graph-based state representations serve as a foundation for advanced applications like Multi-Object Tracking (MOT), offering a comprehensive framework for enhancing the situational awareness and safety of autonomous systems. We validate the effectiveness of our proposed framework through extensive experiments conducted on both synthetic and real-world driving datasets (nuScenes). Our results showcase an improvement in MOTA and a reduction in estimated position errors (MOTP) and identity switches (IDS) for tracked objects using the SAGA-KF. Furthermore, we highlight the capability of such a framework to develop methods that can leverage heterogeneous information (like semantic objects and geometric structures) from various sensing modalities, enabling a more holistic approach to scene understanding and enhancing the safety and effectiveness of autonomous systems.