Unifying VXAI: A Systematic Review and Framework for the Evaluation of Explainable AI

TL;DR

This paper provides a comprehensive review and a unified framework for evaluating explainable AI methods, addressing the lack of standardized metrics and protocols to improve trustworthiness and comparability.

Contribution

It conducts a systematic review of XAI evaluation methods and introduces VXAI, a structured framework for standardized assessment and comparison of XAI explanations.

Findings

Identified 362 relevant publications on XAI evaluation

Aggregated 41 metric groups for explanation assessment

Proposed a three-dimensional categorization scheme for explanations

Abstract

Modern AI systems frequently rely on opaque black-box models, most notably Deep Neural Networks, whose performance stems from complex architectures with millions of learned parameters. While powerful, their complexity poses a major challenge to trustworthiness, particularly due to a lack of transparency. Explainable AI (XAI) addresses this issue by providing human-understandable explanations of model behavior. However, to ensure their usefulness and trustworthiness, such explanations must be rigorously evaluated. Despite the growing number of XAI methods, the field lacks standardized evaluation protocols and consensus on appropriate metrics. To address this gap, we conduct a systematic literature review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and introduce a unified framework for the eValuation of XAI (VXAI). We identify 362 relevant publications and aggregate their contributions into 41 functionally similar metric groups. In addition, we propose a three-dimensional categorization scheme spanning explanation type, evaluation contextuality, and explanation quality desiderata. Our framework provides the most comprehensive and structured overview of VXAI to date. It supports systematic metric selection, promotes comparability across methods, and offers a flexible foundation for future extensions.