Towards Trustworthy and Explainable AI for Perception Models: From Concept to Prototype Vehicle Deployment

TL;DR

This paper presents a Trustworthy AI perception module for autonomous driving that offers robustness, explainability, and calibrated uncertainty, demonstrated through real-time deployment in a prototype vehicle.

Contribution

It introduces a novel Trustworthy AI perception system combining attention-based explanations, uncertainty calibration, and robustness training, with real-world vehicle deployment.

Findings

Faithful saliency maps validate explainability.

Enhanced robustness and calibration of uncertainty estimates.

Real-time trustworthy perception monitoring demonstrated in a prototype vehicle.

Abstract

Deep Neural Networks have become the dominant solution for Autonomous Driving perception, but their opacity conflicts with emerging Trustworthy AI guidelines and complicates safety assurance, debugging, and human oversight. While theoretical frameworks for safe and Explainable AI (XAI) exist, concrete implementations of Trustworthy AI for 3D scene understanding remain scarce. We address this gap by proposing a Trustworthy AI perception module that is remarkably robust, integrates faithful explainability, and calibrated uncertainty estimates. Building on a transformer-based detector, we derive explanation from the attention mechanism at inference time and validate their faithfulness using perturbation-based consistency tests. We further integrate an uncertainty estimation and calibration module, and apply robustness-enhancing training methods. Experiments show faithful saliency behavior, improved robustness, and well-calibrated uncertainty estimates. Finally, we deploy these Trustworthy AI elements in a prototype vehicle and provide an XAI Interface that visualizes documentation artifacts, model uncertainty state, and saliency maps, demonstrating the feasibility of trustworthy perception monitoring in real time. Supplementary materials are available at https://tillbeemelmanns.github.io/trustworthy_ai/ .