Revisiting the Importance of Individual Units in CNNs via Ablation

TL;DR

This paper investigates the role of individual units in CNNs, revealing that while ablating single units doesn't affect overall accuracy, it significantly impacts class-specific performance, highlighting the importance of specialized units.

Contribution

The study demonstrates that individual units are specialized for specific classes and that class selectivity predicts their importance, providing new insights into CNN interpretability.

Findings

Ablating units doesn't reduce overall accuracy.

Ablation impacts class-specific accuracy significantly.

Class selectivity predicts unit importance for classes.

Abstract

We revisit the importance of the individual units in Convolutional Neural Networks (CNNs) for visual recognition. By conducting unit ablation experiments on CNNs trained on large scale image datasets, we demonstrate that, though ablating any individual unit does not hurt overall classification accuracy, it does lead to significant damage on the accuracy of specific classes. This result shows that an individual unit is specialized to encode information relevant to a subset of classes. We compute the correlation between the accuracy drop under unit ablation and various attributes of an individual unit such as class selectivity and weight L1 norm. We confirm that unit attributes such as class selectivity are a poor predictor for impact on overall accuracy as found previously in recent work \cite{morcos2018importance}. However, our results show that class selectivity along with other attributes are good predictors of the importance of one unit to individual classes. We evaluate the impact of random rotation, batch normalization, and dropout to the importance of units to specific classes. Our results show that units with high selectivity play an important role in network classification power at the individual class level. Understanding and interpreting the behavior of these units is necessary and meaningful.