Compositional Explanations of Neurons

TL;DR

This paper introduces a method for explaining neurons in deep models by identifying compositional logical concepts, enabling more precise interpretability and insights into model behavior across vision and language tasks.

Contribution

The paper presents a novel compositional explanation procedure that improves interpretability of neurons and reveals their role in model performance and vulnerabilities.

Findings

Vision neurons learn abstract, coherent visual concepts.

NLI neurons often rely on shallow lexical heuristics.

Interpretable neurons correlate with better task performance.

Abstract

We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. Finally, we show how compositional explanations provide an accessible way for end users to produce simple "copy-paste" adversarial examples that change model behavior in predictable ways.