From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance

TL;DR

This paper introduces a scalable framework to interpret CLIP's internal components, revealing unexpected semantic reliance and spurious correlations, thereby advancing understanding of its decision-making process.

Contribution

The authors adapt attribution patching for CLIP, uncovering how latent components influence predictions and identifying unexpected semantic dependencies and artifacts.

Findings

Uncovered hundreds of surprising components linked to polysemous words and artifacts.

Text embeddings are more robust to spurious correlations than image embeddings.

Case study shows classifiers can amplify hidden shortcuts in skin lesion detection.

Abstract

Transformer-based CLIP models are widely used for text-image probing and feature extraction, making it relevant to understand the internal mechanisms behind their predictions. While recent works show that Sparse Autoencoders (SAEs) yield interpretable latent components, they focus on what these encode and miss how they drive predictions. We introduce a scalable framework that reveals what latent components activate for, how they align with expected semantics, and how important they are to predictions. To achieve this, we adapt attribution patching for instance-wise component attributions in CLIP and highlight key faithfulness limitations of the widely used Logit Lens technique. By combining attributions with semantic alignment scores, we can automatically uncover reliance on components that encode semantically unexpected or spurious concepts. Applied across multiple CLIP variants, our method uncovers hundreds of surprising components linked to polysemous words, compound nouns, visual typography and dataset artifacts. While text embeddings remain prone to semantic ambiguity, they are more robust to spurious correlations compared to linear classifiers trained on image embeddings. A case study on skin lesion detection highlights how such classifiers can amplify hidden shortcuts, underscoring the need for holistic, mechanistic interpretability. We provide code at https://github.com/maxdreyer/attributing-clip.