XAI-Units: Benchmarking Explainability Methods with Unit Tests

TL;DR

This paper introduces XAI-Units, an open-source benchmark for evaluating feature attribution methods in explainable AI, using synthetic datasets and models with known mechanisms to systematically compare their effectiveness.

Contribution

The paper presents a novel benchmark that enables objective, systematic evaluation of FA methods against diverse model behaviors using unit-test-like procedures.

Findings

XAI-Units effectively differentiates FA methods based on model behavior types.

Benchmark reveals strengths and weaknesses of various attribution techniques.

Procedurally generated models facilitate reliable comparison of explainability methods.

Abstract

Feature attribution (FA) methods are widely used in explainable AI (XAI) to help users understand how the inputs of a machine learning model contribute to its outputs. However, different FA models often provide disagreeing importance scores for the same model. In the absence of ground truth or in-depth knowledge about the inner workings of the model, it is often difficult to meaningfully determine which of the different FA methods produce more suitable explanations in different contexts. As a step towards addressing this issue, we introduce the open-source XAI-Units benchmark, specifically designed to evaluate FA methods against diverse types of model behaviours, such as feature interactions, cancellations, and discontinuous outputs. Our benchmark provides a set of paired datasets and models with known internal mechanisms, establishing clear expectations for desirable attribution scores. Accompanied by a suite of built-in evaluation metrics, XAI-Units streamlines systematic experimentation and reveals how FA methods perform against distinct, atomic kinds of model reasoning, similar to unit tests in software engineering. Crucially, by using procedurally generated models tied to synthetic datasets, we pave the way towards an objective and reliable comparison of FA methods.