Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation

TL;DR

This paper explores neighbor Shapley methods for concept-based explanations in deep neural networks, aiming to improve interpretability in critical applications like medical diagnosis and financial analysis.

Contribution

It introduces a novel approach comparing instance-wise and class-wise neighbor Shapley methods for better model interpretability.

Findings

Neighbor Shapley methods enhance explanation quality.

Class-wise approach provides more global insights.

Experimental results demonstrate improved interpretability.

Abstract

Deep neural networks have demonstrated remarkable performance in many data-driven and prediction-oriented applications, and sometimes even perform better than humans. However, their most significant drawback is the lack of interpretability, which makes them less attractive in many real-world applications. When relating to the moral problem or the environmental factors that are uncertain such as crime judgment, financial analysis, and medical diagnosis, it is essential to mine the evidence for the model's prediction (interpret model knowledge) to convince humans. Thus, investigating how to interpret model knowledge is of paramount importance for both academic research and real applications.