The Spotlight Resonance Method: Resolving the Alignment of Embedded Activations

TL;DR

The paper introduces the Spotlight Resonance Method, a novel visualization tool for analyzing the axis alignment of embedded activations in deep learning models, revealing how activation functions influence representational alignment.

Contribution

It presents a versatile visualization technique that evaluates activation distribution around privileged basis vectors, linking functional symmetry breaking to representational alignment.

Findings

Embedded representations tend to be axis-aligned with privileged basis.

Activation functions directly influence the formation of privileged bases.

Examples of grandmother neurons are identified across various networks.

Abstract

Understanding how deep learning models represent data is currently difficult due to the limited number of methodologies available. This paper demonstrates a versatile and novel visualisation tool for determining the axis alignment of embedded data at any layer in any deep learning model. In particular, it evaluates the distribution around planes defined by the network's privileged basis vectors. This method provides both an atomistic and a holistic, intuitive metric for interpreting the distribution of activations across all planes. It ensures that both positive and negative signals contribute, treating the activation vector as a whole. Depending on the application, several variations of this technique are presented, with a resolution scale hyperparameter to probe different angular scales. Using this method, multiple examples are provided that demonstrate embedded representations tend to be axis-aligned with the privileged basis. This is not necessarily the standard basis, and it is found that activation functions directly result in privileged bases. Hence, it provides a direct causal link between functional form symmetry breaking and representational alignment, explaining why representations have a tendency to align with the neuron basis. Therefore, using this method, we begin to answer the fundamental question of what causes the observed tendency of representations to align with neurons. Finally, examples of so-called grandmother neurons are found in a variety of networks.