How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations

TL;DR

This paper investigates how recent multimodal foundation models internally represent text and speech across languages, revealing layer-wise convergence, the importance of length adaptation, and differences in cross-lingual and cross-modal gaps.

Contribution

It provides a detailed analysis of internal representations in multimodal models, highlighting the effects of language, modality, and adaptation techniques on their encoding.

Findings

Cross-modal representations converge over layers except in initial specialized layers.

Length adaptation reduces cross-modal gaps, mainly effective for high-resource languages.

Speech shows larger cross-lingual differences than text.

Abstract

Multimodal foundation models aim to create a unified representation space that abstracts away from surface features like language syntax or modality differences. To investigate this, we study the internal representations of three recent models, analyzing the model activations from semantically equivalent sentences across languages in the text and speech modalities. Our findings reveal that: 1) Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing. 2) Length adaptation is crucial for reducing the cross-modal gap between text and speech, although current approaches' effectiveness is primarily limited to high-resource languages. 3) Speech exhibits larger cross-lingual differences than text. 4) For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.