Quantifying Local Specialization in Deep Neural Networks

TL;DR

This paper introduces methods to quantify local specialization in deep neural networks by analyzing neuron clusters for importance and coherence, revealing meaningful functional groups through graph-based partitioning.

Contribution

It proposes new proxies and statistical techniques to measure neural cluster specialization, enabling automated identification of interpretable neuron groups.

Findings

Graph-based partitioning reveals important neuron groups.

Statistical proxies effectively measure specialization.

Non-runtime analysis can uncover functional clusters.

Abstract

A neural network is locally specialized to the extent that parts of its computational graph (i.e. structure) can be abstractly represented as performing some comprehensible sub-task relevant to the overall task (i.e. functionality). Are modern deep neural networks locally specialized? How can this be quantified? In this paper, we consider the problem of taking a neural network whose neurons are partitioned into clusters, and quantifying how functionally specialized the clusters are. We propose two proxies for this: importance, which reflects how crucial sets of neurons are to network performance; and coherence, which reflects how consistently their neurons associate with features of the inputs. To measure these proxies, we develop a set of statistical methods based on techniques conventionally used to interpret individual neurons. We apply the proxies to partitionings generated by spectrally clustering a graph representation of the network's neurons with edges determined either by network weights or correlations of activations. We show that these partitionings, even ones based only on weights (i.e. strictly from non-runtime analysis), reveal groups of neurons that are important and coherent. These results suggest that graph-based partitioning can reveal local specialization and that statistical methods can be used to automatedly screen for sets of neurons that can be understood abstractly.