ExplainBench: A Benchmark Framework for Local Model Explanations in Fairness-Critical Applications

TL;DR

ExplainBench is an open-source benchmarking framework that systematically evaluates local explanation methods in fairness-critical machine learning applications, promoting reproducibility and accountability.

Contribution

The paper introduces ExplainBench, a standardized, comprehensive toolkit for comparing local explanation techniques in ethically sensitive domains, filling a gap in reproducible evaluation frameworks.

Findings

Different explanation methods exhibit varied fidelity and robustness.

ExplainBench enables reproducible comparison across datasets like COMPAS and UCI Adult.

The framework facilitates understanding of explanation behavior in fairness-sensitive contexts.

Abstract

As machine learning systems are increasingly deployed in high-stakes domains such as criminal justice, finance, and healthcare, the demand for interpretable and trustworthy models has intensified. Despite the proliferation of local explanation techniques, including SHAP, LIME, and counterfactual methods, there exists no standardized, reproducible framework for their comparative evaluation, particularly in fairness-sensitive settings. We introduce ExplainBench, an open-source benchmarking suite for systematic evaluation of local model explanations across ethically consequential datasets. ExplainBench provides unified wrappers for popular explanation algorithms, integrates end-to-end pipelines for model training and explanation generation, and supports evaluation via fidelity, sparsity, and robustness metrics. The framework includes a Streamlit-based graphical interface for interactive exploration and is packaged as a Python module for seamless integration into research workflows. We demonstrate ExplainBench on datasets commonly used in fairness research, such as COMPAS, UCI Adult Income, and LendingClub, and showcase how different explanation methods behave under a shared experimental protocol. By enabling reproducible, comparative analysis of local explanations, ExplainBench advances the methodological foundations of interpretable machine learning and facilitates accountability in real-world AI systems.