Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI

TL;DR

This paper proposes a comprehensive, human-centred evaluation framework for explainable AI, adapting a user-centric approach from recommender systems to holistically assess explanation quality and user experience.

Contribution

It introduces a novel evaluation framework that categorizes explanation properties and metrics, aiming to standardize human-centred XAI evaluation methods.

Findings

Framework integrates explanation aspects and properties.

Categorizes metrics measuring explanation effects.

Aims to standardize human-centred evaluation procedures.

Abstract

While research on explainable AI (XAI) is booming and explanation techniques have proven promising in many application domains, standardised human-centred evaluation procedures are still missing. In addition, current evaluation procedures do not assess XAI methods holistically in the sense that they do not treat explanations' effects on humans as a complex user experience. To tackle this challenge, we propose to adapt the User-Centric Evaluation Framework used in recommender systems: we integrate explanation aspects, summarise explanation properties, indicate relations between them, and categorise metrics that measure these properties. With this comprehensive evaluation framework, we hope to contribute to the human-centred standardisation of XAI evaluation.