Understanding the Importance of Single Directions via Representative Substitution

TL;DR

This paper introduces Representative Substitution (RS) as a new way to understand neural network representations, showing that highly interpretable units are not crucial for generalization, shifting focus to the independence of representations.

Contribution

The work proposes RS as a novel perspective to interpret DNNs, challenging previous assumptions about the importance of interpretable units for generalization.

Findings

Interpretable units have high RS and are not critical for generalization.

RS provides new insights into the independence of neural representations.

Focus should shift from individual units to the relationships among representations.

Abstract

Understanding the internal representations of deep neural networks (DNNs) is crucal to explain their behavior. The interpretation of individual units, which are neurons in MLPs or convolution kernels in convolutional networks, has been paid much attention given their fundamental role. However, recent research (Morcos et al. 2018) presented a counterintuitive phenomenon, which suggests that an individual unit with high class selectivity, called interpretable units, has poor contributions to generalization of DNNs. In this work, we provide a new perspective to understand this counterintuitive phenomenon, which makes sense when we introduce Representative Substitution (RS). Instead of individually selective units with classes, the RS refers to the independence of a unit's representations in the same layer without any annotation. Our experiments demonstrate that interpretable units have high RS which are not critical to network's generalization. The RS provides new insights into the interpretation of DNNs and suggests that we need to focus on the independence and relationship of the representations.