How Important Is a Neuron?

TL;DR

This paper introduces 'conductance', a new method for measuring the importance of hidden units in deep networks, enhancing interpretability beyond input features.

Contribution

The paper proposes conductance as a novel attribution measure for hidden units, supported by theoretical analysis and empirical validation on image and text models.

Findings

Conductance effectively identifies important hidden units.

It provides insights into network internals for different data types.

Supports feature selection and interpretability tasks.

Abstract

The problem of attributing a deep network's prediction to its \emph{input/base} features is well-studied. We introduce the notion of \emph{conductance} to extend the notion of attribution to the understanding the importance of \emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the \emph{flow} of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.