ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

TL;DR

ModalChorus is an interactive system that visualizes and aligns multi-modal embeddings, improving understanding and correction of cross-modal feature misalignments in vision-language models like CLIP.

Contribution

It introduces Modal Fusion Map, a novel dimensionality reduction technique, and an interactive alignment process for better visualization and correction of multi-modal embeddings.

Findings

MFM outperforms t-SNE and MDS in visualizing cross-modal features.

ModalChorus enables intuitive discovery of embedding misalignments.

System improves tasks like zero-shot classification and cross-modal retrieval.

Abstract

Multi-modal embeddings form the foundation for vision-language models, such as CLIP embeddings, the most widely used text-image embeddings. However, these embeddings are vulnerable to subtle misalignment of cross-modal features, resulting in decreased model performance and diminished generalization. To address this problem, we design ModalChorus, an interactive system for visual probing and alignment of multi-modal embeddings. ModalChorus primarily offers a two-stage process: 1) embedding probing with Modal Fusion Map (MFM), a novel parametric dimensionality reduction method that integrates both metric and nonmetric objectives to enhance modality fusion; and 2) embedding alignment that allows users to interactively articulate intentions for both point-set and set-set alignments. Quantitative and qualitative comparisons for CLIP embeddings with existing dimensionality reduction (e.g., t-SNE and MDS) and data fusion (e.g., data context map) methods demonstrate the advantages of MFM in showcasing cross-modal features over common vision-language datasets. Case studies reveal that ModalChorus can facilitate intuitive discovery of misalignment and efficient re-alignment in scenarios ranging from zero-shot classification to cross-modal retrieval and generation.