When more precision is worse: Do people recognize inadequate scene representations in concept-based explainable AI?

TL;DR

This study investigates whether people can recognize when an AI relies on irrelevant features in its explanations, revealing that they often fail to identify such reliance, which impacts trust in AI decisions.

Contribution

It introduces a novel experimental approach to assess human recognition of irrelevant features in AI explanations, highlighting limitations in interpretability.

Findings

Participants favored AI explanations with relevant features.

Participants often did not recognize reliance on irrelevant features.

People may trust flawed AI explanations due to inability to detect irrelevant features.

Abstract

Explainable artificial intelligence (XAI) aims to help uncover flaws in an AI model's internal representations. But do people draw the right conclusions from its explanations? Specifically, do they recognize an AI's inability to distinguish between relevant and irrelevant features? In the present study, a simulated AI classified images of railway trespassers as dangerous or not. To explain which features it has used, other images from the dataset were shown that activate the AI in a similar way. These concept images varied in three relevant features (i.e., a person's distance to the tracks, direction, and action) and in an irrelevant feature (i.e., scene background). When the AI uses a feature in its decision, this feature is retained in the concept images, otherwise the images randomize over it (e.g., same distance, varied backgrounds). Participants rated the AI more favorably when it retained relevant features. For the irrelevant feature, they did not mind in general, and sometimes even preferred it to be retained. This suggests that people may not recognize it when an AI model relies on irrelevant features to make its decisions.