Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

TL;DR

This paper introduces an adversarial prompting method to improve the robustness of text-centric multimodal alignment, effectively handling noise, input permutations, and missing modalities in multimodal models.

Contribution

The study proposes a novel adversarial training approach that enhances the robustness of text-centric multimodal alignment beyond existing methods and pre-trained models.

Findings

Significantly improved robustness against noise and input variations.

Outperforms traditional robust training and pre-trained models in experiments.

Enhances adaptability of multimodal representations in real-world scenarios.

Abstract

Converting different modalities into generalized text, which then serves as input prompts for large language models (LLMs), is a common approach for aligning multimodal models, particularly when pairwise data is limited. Text-centric alignment method leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation, thereby enabling downstream models to effectively interpret various modal inputs. This study evaluates the quality and robustness of multimodal representations in the face of noise imperfections, dynamic input order permutations, and missing modalities, revealing that current text-centric alignment methods can compromise downstream robustness. To address this issue, we propose a new text-centric adversarial training approach that significantly enhances robustness compared to traditional robust training methods and pre-trained multimodal foundation models. Our findings underscore the potential of this approach to improve the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications.