People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI

TL;DR

This study investigates how people attribute purpose to autonomous vehicle behavior using cognitive science insights, revealing that teleological explanations are perceived as higher quality, informing better design of explainable AI in safety-critical domains.

Contribution

The paper introduces a framework of explanatory modes based on cognitive science and provides empirical evidence on how these modes affect explanation perception in autonomous driving.

Findings

Teleological explanations are rated higher in quality than counterfactual explanations.

Perceived teleology strongly predicts explanation quality.

The study provides a new dataset, HEADD, for analyzing explanations in autonomous driving.

Abstract

It is often argued that effective human-centered explainable artificial intelligence (XAI) should resemble human reasoning. However, empirical investigations of how concepts from cognitive science can aid the design of XAI are lacking. Based on insights from cognitive science, we propose a framework of explanatory modes to analyze how people frame explanations, whether mechanistic, teleological, or counterfactual. Using the complex safety-critical domain of autonomous driving, we conduct an experiment consisting of two studies on (i) how people explain the behavior of a vehicle in 14 unique scenarios (N1=54) and (ii) how they perceive these explanations (N2=382), curating the novel Human Explanations for Autonomous Driving Decisions (HEADD) dataset. Our main finding is that participants deem teleological explanations significantly better quality than counterfactual ones, with perceived teleology being the best predictor of perceived quality. Based on our results, we argue that explanatory modes are an important axis of analysis when designing and evaluating XAI and highlight the need for a principled and empirically grounded understanding of the cognitive mechanisms of explanation. The HEADD dataset and our code are available at: https://datashare.ed.ac.uk/handle/10283/8930.