Exploring SAIG Methods for an Objective Evaluation of XAI

TL;DR

This paper reviews and analyzes SAIG methods for objectively evaluating XAI techniques by generating artificial ground truths, introduces a taxonomy, and highlights the need for standardization in evaluation approaches.

Contribution

It provides the first comprehensive review of SAIG methods, proposing a taxonomy and analyzing their key features to advance XAI evaluation research.

Findings

Lack of consensus on effective XAI evaluation methods

Seven key features distinguish different SAIG approaches

Highlighting the need for standardization in XAI evaluation

Abstract

The evaluation of eXplainable Artificial Intelligence (XAI) methods is a rapidly growing field, characterized by a wide variety of approaches. This diversity highlights the complexity of the XAI evaluation, which, unlike traditional AI assessment, lacks a universally correct ground truth for the explanation, making objective evaluation challenging. One promising direction to address this issue involves the use of what we term Synthetic Artificial Intelligence Ground truth (SAIG) methods, which generate artificial ground truths to enable the direct evaluation of XAI techniques. This paper presents the first review and analysis of SAIG methods. We introduce a novel taxonomy to classify these approaches, identifying seven key features that distinguish different SAIG methods. Our comparative study reveals a concerning lack of consensus on the most effective XAI evaluation techniques, underscoring the need for further research and standardization in this area.