FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

TL;DR

This paper introduces FLiP models to interpret and analyze multilingual and multimodal sentence embeddings, revealing biases and intrinsic properties without relying on downstream tasks.

Contribution

The paper proposes factorized linear projection models that effectively recover lexical content and diagnose biases in pretrained sentence embeddings across modalities and languages.

Findings

FLiP recalls over 75% of lexical content from embeddings.

FLiP outperforms existing non-factorized baselines.

Uncovers modality and language biases in sentence encoders.

Abstract

This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from multilingual (LaBSE), multimodal (SONAR) and API-based (Gemini) sentence embedding spaces in several high- and mid-resource languages. We show that FLiP can recall more than 75% of lexical content from the embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP.