Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis

TL;DR

This study evaluates how well users understand and predict black-box model behavior using post-hoc explanations, revealing factors that influence comprehensibility and suggesting design improvements for explainability tools.

Contribution

It provides an empirical user-centered analysis of LIME and SHAP, highlighting how explanations, counterfactuals, and misclassifications affect user understanding.

Findings

SHAP explanations are less comprehensible near decision boundaries

Counterfactual explanations improve user understanding

Misclassifications can enhance predictability

Abstract

Post-hoc explainability methods aim to clarify predictions of black-box machine learning models. However, it is still largely unclear how well users comprehend the provided explanations and whether these increase the users ability to predict the model behavior. We approach this question by conducting a user study to evaluate comprehensibility and predictability in two widely used tools: LIME and SHAP. Moreover, we investigate the effect of counterfactual explanations and misclassifications on users ability to understand and predict the model behavior. We find that the comprehensibility of SHAP is significantly reduced when explanations are provided for samples near a model's decision boundary. Furthermore, we find that counterfactual explanations and misclassifications can significantly increase the users understanding of how a machine learning model is making decisions. Based on our findings, we also derive design recommendations for future post-hoc explainability methods with increased comprehensibility and predictability.