On the Road to Clarity: Exploring Explainable AI for World Models in a Driver Assistance System

TL;DR

This paper advances explainable AI techniques for autonomous driving by developing transparent models for VAEs and analyzing internal dynamics of prediction networks, enhancing safety and interpretability.

Contribution

It introduces a refined coarse graining method for VAEs, a new visualization technique for convolutional layers, and explanation methods for prediction networks in autonomous driving.

Findings

Performance of the transparent VAE backbone is comparable to black box VAEs.

The feature map visualization helps identify causes of poor reconstruction.

Analysis of the VAE-LSTM model demonstrates potential for safer traffic prediction.

Abstract

In Autonomous Driving (AD) transparency and safety are paramount, as mistakes are costly. However, neural networks used in AD systems are generally considered black boxes. As a countermeasure, we have methods of explainable AI (XAI), such as feature relevance estimation and dimensionality reduction. Coarse graining techniques can also help reduce dimensionality and find interpretable global patterns. A specific coarse graining method is Renormalization Groups from statistical physics. It has previously been applied to Restricted Boltzmann Machines (RBMs) to interpret unsupervised learning. We refine this technique by building a transparent backbone model for convolutional variational autoencoders (VAE) that allows mapping latent values to input features and has performance comparable to trained black box VAEs. Moreover, we propose a custom feature map visualization technique to analyze the internal convolutional layers in the VAE to explain internal causes of poor reconstruction that may lead to dangerous traffic scenarios in AD applications. In a second key contribution, we propose explanation and evaluation techniques for the internal dynamics and feature relevance of prediction networks. We test a long short-term memory (LSTM) network in the computer vision domain to evaluate the predictability and in future applications potentially safety of prediction models. We showcase our methods by analyzing a VAE-LSTM world model that predicts pedestrian perception in an urban traffic situation.