Linear Spaces of Meanings: Compositional Structures in Vision-Language Models

TL;DR

This paper explores the geometric and probabilistic structures of embeddings in vision-language models, demonstrating how simple linear operations can improve interpretability and task performance.

Contribution

It introduces a geometric framework for understanding compositionality in VLM embeddings and empirically shows the effectiveness of linear operations for various vision-language tasks.

Findings

Linear algebraic operations enable interpretable manipulation of embeddings

Compositional structures improve performance in classification and retrieval

Embedding vectors can be approximated as combinations of 'ideal words'

Abstract

We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs). Traditionally, compositionality has been associated with algebraic operations on embeddings of words from a pre-existing vocabulary. In contrast, we seek to approximate representations from an encoder as combinations of a smaller set of vectors in the embedding space. These vectors can be seen as "ideal words" for generating concepts directly within the embedding space of the model. We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these compositional structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice. Finally, we empirically explore these structures in CLIP's embeddings and we evaluate their usefulness for solving different vision-language tasks such as classification, debiasing, and retrieval. Our results show that simple linear algebraic operations on embedding vectors can be used as compositional and interpretable methods for regulating the behavior of VLMs.