Explainable deep learning improves human mental models of self-driving cars

TL;DR

This paper presents a real-world deployment of explainable deep learning in self-driving cars, demonstrating that concept-based explanations improve human understanding and prediction of the vehicle's behavior without compromising performance.

Contribution

Introduction of CW-Net, a novel concept-grounded explanation method for autonomous driving, validated in a real car setting to enhance human mental models.

Findings

Explanations improve human prediction accuracy of the car's behavior.

CW-Net explanations are causally faithful and do not reduce driving performance.

First real-world demonstration of explainable AI aiding autonomous vehicle understanding.

Abstract

Self-driving cars increasingly rely on deep neural networks to achieve human-like driving. The opacity of such black-box planners makes it challenging for the human behind the wheel to accurately anticipate when they will fail, with potentially catastrophic consequences. While research into interpreting these systems has surged, most of it is confined to simulations or toy setups due to the difficulty of real-world deployment, leaving the practical utility of such techniques unknown. Here, we introduce the Concept-Wrapper Network (CW-Net), a method for explaining the behavior of machine-learning-based planners by grounding their reasoning in human-interpretable concepts. We deploy CW-Net on a real self-driving car and show that the resulting explanations improve the human driver's mental model of the car, allowing them to better predict its behavior. To our knowledge, this is the first demonstration that explainable deep learning integrated into self-driving cars can be both understandable and useful in a realistic deployment setting. CW-Net accomplishes this level of intelligibility while providing explanations which are causally faithful and do not sacrifice driving performance. Overall, our study establishes a general pathway to interpretability for autonomous agents by way of concept-based explanations, which could help make them more transparent and safe.