BONES: a Benchmark fOr Neural Estimation of Shapley values

TL;DR

BONES is a comprehensive benchmark suite designed to standardize and facilitate the evaluation and comparison of neural estimators for Shapley Values in explainable AI, covering datasets, models, metrics, and visualizations.

Contribution

It introduces BONES, an open-source benchmark with standardized tools, datasets, and evaluation metrics for neural Shapley Value estimators, enhancing reproducibility and comparability.

Findings

Demonstrated BONES effectiveness on tabular and image data

Provided baseline results for neural Shapley estimators

Facilitated easier evaluation and visualization of explainability models

Abstract

Shapley Values are concepts established for eXplainable AI. They are used to explain black-box predictive models by quantifying the features' contributions to the model's outcomes. Since computing the exact Shapley Values is known to be computationally intractable on real-world datasets, neural estimators have emerged as alternative, more scalable approaches to get approximated Shapley Values estimates. However, experiments with neural estimators are currently hard to replicate as algorithm implementations, explainer evaluators, and results visualizations are neither standardized nor promptly usable. To bridge this gap, we present BONES, a new benchmark focused on neural estimation of Shapley Value. It provides researchers with a suite of state-of-the-art neural and traditional estimators, a set of commonly used benchmark datasets, ad hoc modules for training black-box models, as well as specific functions to easily compute the most popular evaluation metrics and visualize results. The purpose is to simplify XAI model usage, evaluation, and comparison. In this paper, we showcase BONES results and visualizations for XAI model benchmarking on both tabular and image data. The open-source library is available at the following link: https://github.com/DavideNapolitano/BONES.