Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings

TL;DR

This paper investigates the geometry of vision-language embeddings using the Iso-Energy Assumption and introduces an autoencoder that reveals how cross-modal alignment is structured, enabling better interpretability and manipulation of models.

Contribution

The authors propose a novel framework based on the Iso-Energy Assumption and an aligned sparse autoencoder to analyze and interpret the geometry of vision-language embeddings.

Findings

Sparse bimodal atoms encode the entire cross-modal alignment.

Unimodal atoms explain the modality gap and act as biases.

Removing unimodal atoms collapses the gap without affecting performance.

Abstract

Vision-language models (VLMs) align images and text with remarkable success, yet the geometry of their shared embedding space remains poorly understood. To probe this geometry, we begin from the Iso-Energy Assumption, which exploits cross-modal redundancy: a concept that is truly shared should exhibit the same average energy across modalities. We operationalize this assumption with an Aligned Sparse Autoencoder (SAE) that encourages energy consistency during training while preserving reconstruction. We find that this inductive bias changes the SAE solution without harming reconstruction, giving us a representation that serves as a tool for geometric analysis. Sanity checks on controlled data with known ground truth confirm that alignment improves when Iso-Energy holds and remains neutral when it does not. Applied to foundational VLMs, our framework reveals a clear structure with practical consequences: (i) sparse bimodal atoms carry the entire cross-modal alignment signal; (ii) unimodal atoms act as modality-specific biases and fully explain the modality gap; (iii) removing unimodal atoms collapses the gap without harming performance; (iv) restricting vector arithmetic to the bimodal subspace yields in-distribution edits and improved retrieval. These findings suggest that the right inductive bias can both preserve model fidelity and render the latent geometry interpretable and actionable.