Interpreting the Second-Order Effects of Neurons in CLIP

TL;DR

This paper introduces the 'second-order lens' to interpret CLIP neurons by analyzing effects through attention heads, revealing their selectivity and polysemantic nature, and applying this understanding to adversarial example generation and zero-shot segmentation.

Contribution

The paper presents a novel second-order analysis method for interpreting CLIP neurons, uncovering their selectivity and polysemy, and demonstrates practical applications in adversarial attacks and segmentation.

Findings

Neuron effects are highly selective, impacting less than 2% of images.

Each neuron effect can be approximated by a single text-image direction.

Neurons exhibit polysemantic behavior, representing multiple unrelated concepts.

Abstract

We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.