LUCID-SAE: Learning Unified Vision-Language Sparse Codes for Interpretable Concept Discovery

TL;DR

LUCID introduces a unified vision-language autoencoder that learns shared and private features for images and text, enabling interpretable, cross-modal concept discovery without labels.

Contribution

It proposes a novel shared latent dictionary for vision and language, with an alignment method that improves interpretability and transferability of features across modalities.

Findings

Shared features support patch-level grounding

Establish cross-modal neuron correspondence

Capture diverse semantic categories beyond objects

Abstract

Sparse autoencoders (SAEs) offer a natural path toward comparable explanations across different representation spaces. However, current SAEs are trained per modality, producing dictionaries whose features are not directly understandable and whose explanations do not transfer across domains. In this study, we introduce LUCID (Learning Unified vision-language sparse Codes for Interpretable concept Discovery), a unified vision-language sparse autoencoder that learns a shared latent dictionary for image patch and text token representations, while reserving private capacity for modality-specific details. We achieve feature alignment by coupling the shared codes with a learned optimal transport matching objective without the need of labeling. LUCID yields interpretable shared features that support patch-level grounding, establish cross-modal neuron correspondence, and enhance robustness against the concept clustering problem in similarity-based evaluation. Leveraging the alignment properties, we develop an automated dictionary interpretation pipeline based on term clustering without manual observations. Our analysis reveals that LUCID's shared features capture diverse semantic categories beyond objects, including actions, attributes, and abstract concepts, demonstrating a comprehensive approach to interpretable multimodal representations.