Learning Multi-Modal Self-Awareness Models for Autonomous Vehicles from Human Driving

TL;DR

This paper introduces a multi-modal self-awareness modeling approach for autonomous vehicles using synchronized sensor data, enabling improved anomaly detection and decision-making.

Contribution

It presents a novel method combining coupled Dynamic Bayesian Networks to learn and correlate multi-modal sensor data for autonomous vehicle self-awareness.

Findings

Effective multi-modal correlation discovered in sensor data

Enhanced anomaly detection capabilities demonstrated

Potential for improved autonomous decision-making

Abstract

This paper presents a novel approach for learning self-awareness models for autonomous vehicles. The proposed technique is based on the availability of synchronized multi-sensor dynamic data related to different maneuvering tasks performed by a human operator. It is shown that different machine learning approaches can be used to first learn single modality models using coupled Dynamic Bayesian Networks; such models are then correlated at event level to discover contextual multi-modal concepts. In the presented case, visual perception and localization are used as modalities. Cross-correlations among modalities in time is discovered from data and are described as probabilistic links connecting shared and private multi-modal DBNs at the event (discrete) level. Results are presented on experiments performed on an autonomous vehicle, highlighting potentiality of the proposed approach to allow anomaly detection and autonomous decision making based on learned self-awareness models.