How Important Is a Neuron?
Kedar Dhamdhere, Mukund Sundararajan, Qiqi Yan

TL;DR
This paper introduces 'conductance', a new method for measuring the importance of hidden units in deep networks, enhancing interpretability beyond input features.
Contribution
The paper proposes conductance as a novel attribution measure for hidden units, supported by theoretical analysis and empirical validation on image and text models.
Findings
Conductance effectively identifies important hidden units.
It provides insights into network internals for different data types.
Supports feature selection and interpretability tasks.
Abstract
The problem of attributing a deep network's prediction to its \emph{input/base} features is well-studied. We introduce the notion of \emph{conductance} to extend the notion of attribution to the understanding the importance of \emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the \emph{flow} of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExplainable Artificial Intelligence (XAI) · Adversarial Robustness in Machine Learning · Machine Learning in Materials Science
