CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

TL;DR

This paper introduces a cognitive user model for understanding how humans interpret AI explanations, focusing on reasoning strategies with structured data to improve XAI usability.

Contribution

It develops a cognitive modeling approach based on reasoning strategies, providing insights into human understanding of AI explanations and guiding future XAI design.

Findings

Models better fit human decisions than baseline proxies.

Identifies effective and ineffective reasoning strategies.

Enables hypothesis testing without costly human studies.

Abstract

Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.