A holistic perception system of internal and external monitoring for ground autonomous vehicles: AutoTRUST paradigm

TL;DR

This paper presents a comprehensive perception system for autonomous vehicles that integrates internal occupant monitoring and external environment sensing using advanced AI techniques, validated on a real electric vehicle.

Contribution

It introduces a novel holistic perception framework combining multi-modal internal and external monitoring with AI-driven analysis, deployed on a real vehicle in a European research project.

Findings

Enhanced perception accuracy and efficiency demonstrated in real-world tests.

Successful integration of multi-modal sensors and AI models in a vehicle environment.

Improved occupant and environment monitoring capabilities.

Abstract

This paper introduces a holistic perception system for internal and external monitoring of autonomous vehicles, with the aim of demonstrating a novel AI-leveraged self-adaptive framework of advanced vehicle technologies and solutions that optimize perception and experience on-board. Internal monitoring system relies on a multi-camera setup designed for predicting and identifying driver and occupant behavior through facial recognition, exploiting in addition a large language model as virtual assistant. Moreover, the in-cabin monitoring system includes AI-empowered smart sensors that measure air-quality and perform thermal comfort analysis for efficient on and off-boarding. On the other hand, external monitoring system perceives the surrounding environment of vehicle, through a LiDAR-based cost-efficient semantic segmentation approach, that performs highly accurate and efficient super-resolution on low-quality raw 3D point clouds. The holistic perception framework is developed in the context of EU's Horizon Europe programm AutoTRUST, and has been integrated and deployed on a real electric vehicle provided by ALKE. Experimental validation and evaluation at the integration site of Joint Research Centre at Ispra, Italy, highlights increased performance and efficiency of the modular blocks of the proposed perception architecture.