Linear Explanations for Individual Neurons

TL;DR

This paper introduces a linear explanation method for individual neurons in neural networks, revealing that understanding neurons requires analyzing their entire activation range rather than just their highest activations.

Contribution

The paper proposes a novel linear explanation approach for neurons and introduces an automatic evaluation method using simulation to assess explanation quality.

Findings

High activations account for a small causal effect.

Lower activations are diverse and not predictable from high activations.

Linear explanations effectively capture neuron behavior across activation ranges.

Abstract

In recent years many methods have been developed to understand the internal workings of neural networks, often by describing the function of individual neurons in the model. However, these methods typically only focus on explaining the very highest activations of a neuron. In this paper we show this is not sufficient, and that the highest activation range is only responsible for a very small percentage of the neuron's causal effect. In addition, inputs causing lower activations are often very different and can't be reliably predicted by only looking at high activations. We propose that neurons should instead be understood as a linear combination of concepts, and develop an efficient method for producing these linear explanations. In addition, we show how to automatically evaluate description quality using simulation, i.e. predicting neuron activations on unseen inputs in vision setting.