Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning

TL;DR

This paper introduces Sparse CLIP, a training method that creates sparse, interpretable, and high-performing vision-language representations, challenging the idea that interpretability must compromise accuracy.

Contribution

The authors propose integrating sparsity directly into CLIP training to enhance interpretability without sacrificing downstream performance or multimodal capabilities.

Findings

Sparse CLIP maintains strong downstream task performance.

Sparse representations improve interpretability and concept alignment.

Multimodal capabilities are preserved in sparse representations.

Abstract

Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in vision-language representation learning, powering diverse downstream tasks and serving as the default vision backbone in multimodal large language models (MLLMs). Despite its success, CLIP's dense and opaque latent representations pose significant interpretability challenges. A common assumption is that interpretability and performance are in tension: enforcing sparsity during training degrades accuracy, motivating recent post-hoc approaches such as Sparse Autoencoders (SAEs). However, these post-hoc approaches often suffer from degraded downstream performance and loss of CLIP's inherent multimodal capabilities, with most learned features remaining unimodal. We propose a simple yet effective approach that integrates sparsity directly into CLIP training, yielding representations that are both interpretable and performant. Compared to SAEs, our Sparse CLIP representations preserve strong downstream task performance, achieve superior interpretability, and retain multimodal capabilities. We show that multimodal sparse features enable straightforward semantic concept alignment and reveal training dynamics of how cross-modal knowledge emerges. Finally, as a proof of concept, we train a vision-language model on sparse CLIP representations that enables interpretable, vision-based steering capabilities. Our findings challenge conventional wisdom that interpretability requires sacrificing accuracy and demonstrate that interpretability and performance can be co-optimized, offering a promising design principle for future models.