Explaining How Visual, Textual and Multimodal Encoders Share Concepts

TL;DR

This paper introduces new tools for comparing visual, textual, and multimodal encoders based on sparse autoencoder features, revealing shared representations and the influence of text pretraining across models.

Contribution

It proposes a novel quantitative indicator for cross-modality model comparison and a measure for shared features, enabling comprehensive analysis of different encoder types.

Findings

Visual features in VLMs are shared with text encoders.

Models trained in multimodal contexts share more representations.

The new tools facilitate revisiting and understanding encoder similarities.

Abstract

Sparse autoencoders (SAEs) have emerged as a powerful technique for extracting human-interpretable features from neural networks activations. Previous works compared different models based on SAE-derived features but those comparisons have been restricted to models within the same modality. We propose a novel indicator allowing quantitative comparison of models across SAE features, and use it to conduct a comparative study of visual, textual and multimodal encoders. We also propose to quantify the Comparative Sharedness of individual features between different classes of models. With these two new tools, we conduct several studies on 21 encoders of the three types, with two significantly different sizes, and considering generalist and domain specific datasets. The results allow to revisit previous studies at the light of encoders trained in a multimodal context and to quantify to which extent all these models share some representations or features. They also suggest that visual features that are specific to VLMs among vision encoders are shared with text encoders, highlighting the impact of text pretraining. The code is available at https://github.com/CEA-LIST/SAEshareConcepts