XEQ Scale for Evaluating XAI Experience Quality

TL;DR

This paper introduces the XAI Experience Quality (XEQ) Scale, a comprehensive tool for evaluating user-centered XAI experiences across multiple dimensions, validated through expert input and a large-scale pilot study.

Contribution

The paper presents the development and validation of the XEQ Scale, a novel multi-dimensional framework for assessing the quality of XAI experiences beyond single-shot metrics.

Findings

XEQ Scale effectively measures learning, utility, fulfilment, and engagement.

Validation shows strong evidence of the scale's reliability and discriminant validity.

XEQ Scale advances XAI evaluation by capturing holistic user experiences.

Abstract

Explainable Artificial Intelligence (XAI) aims to improve the transparency of autonomous decision-making through explanations. Recent literature has emphasised users' need for holistic "multi-shot" explanations and personalised engagement with XAI systems. We refer to this user-centred interaction as an XAI Experience. Despite advances in creating XAI experiences, evaluating them in a user-centred manner has remained challenging. In response, we developed the XAI Experience Quality (XEQ) Scale. XEQ quantifies the quality of experiences across four dimensions: learning, utility, fulfilment and engagement. These contributions extend the state-of-the-art of XAI evaluation, moving beyond the one-dimensional metrics frequently developed to assess single-shot explanations. This paper presents the XEQ scale development and validation process, including content validation with XAI experts, and discriminant and construct validation through a large-scale pilot study. Our pilot study results offer strong evidence that establishes the XEQ Scale as a comprehensive framework for evaluating user-centred XAI experiences.