A quantitative analysis of semantic information in deep representations of text and images

TL;DR

This paper investigates how semantic information is distributed and aligned across different deep representations of text and images using an information-theoretic measure, revealing layer-specific and cross-modal predictability patterns.

Contribution

It introduces the use of Information Imbalance to quantify semantic information distribution and predictability across models, languages, and modalities.

Findings

Semantic information is spread across many tokens and layers.

English representations are more predictive than other languages.

Semantic information concentrates in specific layers depending on modality.

Abstract

It was recently observed that the representations of different models that process identical or semantically related inputs tend to align. We analyze this phenomenon using the Information Imbalance, an asymmetric rank-based measure that quantifies the capability of a representation to predict another, providing a proxy of the cross-entropy which can be computed efficiently in high-dimensional spaces. By measuring the Information Imbalance between representations generated by DeepSeek-V3 processing translations, we find that semantic information is spread across many tokens, and that semantic predictability is strongest in a set of central layers of the network, robust across six language pairs. We measure clear information asymmetries: English representations are systematically more predictive than those of other languages, and DeepSeek-V3 representations are more predictive of those in a smaller model such as Llama3-8b than the opposite. In the visual domain, we observe that semantic information concentrates in middle layers for autoregressive models and in final layers for encoder models, and these same layers yield the strongest cross-modal predictability with textual representations of image captions. Our results support the hypothesis of semantic convergence across languages, modalities, and architectures, while showing that directed predictability between representations varies strongly with layer-depth, model scale, and language.