How to Answer Why -- Evaluating the Explanations of AI Through Mental Model Analysis

TL;DR

This paper investigates how to effectively evaluate AI explanations by analyzing users' mental models, proposing methods to assess the validity of mental models and integrating cognitive tutoring techniques for human-centered AI evaluation.

Contribution

It introduces a novel approach to evaluate explainable AI by analyzing mental models and proposes an exemplary method combining mental model elicitation with cognitive tutoring.

Findings

Mental models are suitable for empirical evaluation of AI explanations.

Proposed methods improve understanding of user-AI mental model alignment.

Integration of cognitive tutoring enhances mental model analysis.

Abstract

To achieve optimal human-system integration in the context of user-AI interaction it is important that users develop a valid representation of how AI works. In most of the everyday interaction with technical systems users construct mental models (i.e., an abstraction of the anticipated mechanisms a system uses to perform a given task). If no explicit explanations are provided by a system (e.g. by a self-explaining AI) or other sources (e.g. an instructor), the mental model is typically formed based on experiences, i.e. the observations of the user during the interaction. The congruence of this mental model and the actual systems functioning is vital, as it is used for assumptions, predictions and consequently for decisions regarding system use. A key question for human-centered AI research is therefore how to validly survey users' mental models. The objective of the present research is to identify suitable elicitation methods for mental model analysis. We evaluated whether mental models are suitable as an empirical research method. Additionally, methods of cognitive tutoring are integrated. We propose an exemplary method to evaluate explainable AI approaches in a human-centered way.