Towards Utilising a Range of Neural Activations for Comprehending Representational Associations

TL;DR

This paper investigates the informational value of non-extremal neural activations in deep networks, demonstrating that analyzing a range of activation levels reveals complex representations and can improve model robustness against confounding factors.

Contribution

It introduces a novel approach to study mid-level neural activations, showing their usefulness in understanding representations and mitigating confounding concepts in neural networks.

Findings

Mid-level activations contain valuable information beyond maximal responses.

Analyzing a range of activations helps identify confounding concepts.

Method improves model robustness by curating data based on non-maximal activations.

Abstract

Recent efforts to understand intermediate representations in deep neural networks have commonly attempted to label individual neurons and combinations of neurons that make up linear directions in the latent space by examining extremal neuron activations and the highest direction projections. In this paper, we show that this approach, although yielding a good approximation for many purposes, fails to capture valuable information about the behaviour of a representation. Neural network activations are generally dense, and so a more complex, but realistic scenario is that linear directions encode information at various levels of stimulation. We hypothesise that non-extremal level activations contain complex information worth investigating, such as statistical associations, and thus may be used to locate confounding human interpretable concepts. We explore the value of studying a range of neuron activations by taking the case of mid-level output neuron activations and demonstrate on a synthetic dataset how they can inform us about aspects of representations in the penultimate layer not evident through analysing maximal activations alone. We use our findings to develop a method to curate data from mid-range logit samples for retraining to mitigate spurious correlations, or confounding concepts in the penultimate layer, on real benchmark datasets. The success of our method exemplifies the utility of inspecting non-maximal activations to extract complex relationships learned by models.