Robust Latent Representation Tuning for Image-text Classification

TL;DR

This paper introduces a robust latent representation tuning method for large multimodal models that enhances their ability to perform well even when one modality is missing, by maximizing modality correlation and refining common semantics.

Contribution

The paper proposes a novel latent translation and fusion framework that maintains pre-trained model capabilities while improving robustness to missing modalities.

Findings

Effective in handling missing modality scenarios

Maintains high performance across multiple datasets

Enhances modality correlation and semantic refinement

Abstract

Large models have demonstrated exceptional generalization capabilities in computer vision and natural language processing. Recent efforts have focused on enhancing these models with multimodal processing abilities. However, addressing the challenges posed by scenarios where one modality is absent remains a significant hurdle. In response to this issue, we propose a robust latent representation tuning method for large models. Specifically, our approach introduces a modality latent translation module to maximize the correlation between modalities, resulting in a robust representation. Following this, a newly designed fusion module is employed to facilitate information interaction between the modalities. Within this framework, common semantics are refined during training, and robust performance is achieved even in the absence of one modality. Importantly, our method maintains the frozen state of the image and text foundation models to preserve their capabilities acquired through large-scale pretraining. We conduct experiments on several public datasets, and the results underscore the effectiveness of our proposed method.