Enhance the Robustness of Text-Centric Multimodal Alignments

TL;DR

This paper introduces a new text-centric method to improve the robustness of multimodal alignments, especially under challenging conditions like missing or noisy data, enhancing the reliability of multimodal models.

Contribution

The study proposes a novel text-centric approach that outperforms previous methods in robustness across various modalities and scenarios.

Findings

The new approach improves robustness against missing modalities.

It maintains performance under noisy and incomplete data.

Outperforms previous methods in diverse settings.

Abstract

Converting different modalities into general text, serving as input prompts for large language models (LLMs), is a common method to align multimodal models when there is limited pairwise data. This text-centric approach leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation. This enables downstream models to effectively interpret various modal inputs. This study assesses the quality and robustness of multimodal representations in the presence of missing entries, noise, or absent modalities, revealing that current text-centric alignment methods compromise downstream robustness. To address this issue, we propose a new text-centric approach that achieves superior robustness compared to previous methods across various modalities in different settings. Our findings highlight the potential of this approach to enhance the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications.