Neuron Shapley: Discovering the Responsible Neurons
Amirata Ghorbani, James Zou

TL;DR
Neuron Shapley introduces a novel method to quantify individual neuron contributions in deep networks, effectively identifying critical filters that influence predictions, biases, and vulnerabilities, thereby enhancing interpretability and model repair.
Contribution
This work presents Neuron Shapley, a new framework that accounts for neuron interactions to identify important neurons, along with a multi-arm bandit algorithm for efficient Shapley value estimation.
Findings
Removing top filters destroys Inception-v3 accuracy.
Filters responsible for biases and adversarial vulnerabilities identified.
Neuron Shapley outperforms activation-based importance methods.
Abstract
We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network. By accounting for interactions across neurons, Neuron Shapley is more effective in identifying important filters compared to common approaches based on activation patterns. Interestingly, removing just 30 filters with the highest Shapley scores effectively destroys the prediction accuracy of Inception-v3 on ImageNet. Visualization of these few critical filters provides insights into how the network functions. Neuron Shapley is a flexible framework and can be applied to identify responsible neurons in many tasks. We illustrate additional applications of identifying filters that are responsible for biased prediction in facial recognition and filters that are vulnerable to adversarial attacks. Removing these filters is a quick way to repair…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Explainable Artificial Intelligence (XAI) · Cell Image Analysis Techniques
MethodsRepair · Average Pooling · Auxiliary Classifier · 1x1 Convolution · RMSProp · Inception-v3 Module · Max Pooling · Softmax · Convolution · Dropout
