Towards Autonomous Driving: a Multi-Modal 360$^{\circ}$ Perception Proposal

TL;DR

This paper introduces a multi-modal 360-degree perception framework for autonomous vehicles that combines CNNs, LiDAR, and tracking algorithms to improve 3D object detection and tracking accuracy in real-world driving scenarios.

Contribution

It proposes a novel sensor fusion architecture integrating CNN-based segmentation, LiDAR-image association, PointNet for 3D bounding boxes, and Kalman filtering for tracking.

Findings

System achieves high accuracy in 3D object detection.

Framework performs reliably in real autonomous driving tests.

Sensor fusion enhances perception robustness.

Abstract

In this paper, a multi-modal 360$^{\circ}$ framework for 3D object detection and tracking for autonomous vehicles is presented. The process is divided into four main stages. First, images are fed into a CNN network to obtain instance segmentation of the surrounding road participants. Second, LiDAR-to-image association is performed for the estimated mask proposals. Then, the isolated points of every object are processed by a PointNet ensemble to compute their corresponding 3D bounding boxes and poses. Lastly, a tracking stage based on Unscented Kalman Filter is used to track the agents along time. The solution, based on a novel sensor fusion configuration, provides accurate and reliable road environment detection. A wide variety of tests of the system, deployed in an autonomous vehicle, have successfully assessed the suitability of the proposed perception stack in a real autonomous driving application.