Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning

TL;DR

This paper introduces cross-modal attribute insertions as a realistic perturbation method to evaluate the robustness of vision-and-language models, revealing significant performance drops and emphasizing the importance of multimodal data augmentation.

Contribution

It proposes a novel, controllable, and task-agnostic method for inserting visual attributes into text to assess model robustness in multimodal tasks.

Findings

State-of-the-art models' performance drops by 15-20% with cross-modal insertions.

Crowd-sourced annotations show higher quality augmentations than text-only methods.

The approach is modular, controllable, and applicable across different tasks.

Abstract

The robustness of multimodal deep learning models to realistic changes in the input text is critical for their applicability to important tasks such as text-to-image retrieval and cross-modal entailment. To measure robustness, several existing approaches edit the text data, but do so without leveraging the cross-modal information present in multimodal data. Information from the visual modality, such as color, size, and shape, provide additional attributes that users can include in their inputs. Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., "girl on a chair" to "little girl on a wooden chair"). Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic. We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and cross-modal entailment to perform poorly, resulting in relative drops of 15% in MRR and 20% in $F_1$ score, respectively. Crowd-sourced annotations demonstrate that cross-modal insertions lead to higher quality augmentations for multimodal data than augmentations using text-only data, and are equivalent in quality to original examples. We release the code to encourage robustness evaluations of deep vision-and-language models: https://github.com/claws-lab/multimodal-robustness-xmai.