Unaligning Everything: Or Aligning Any Text to Any Image in Multimodal Models

TL;DR

This paper demonstrates that multimodal models with shared embedding spaces are vulnerable to adversarial attacks that can unnoticeably align any text to any image, exposing a significant security flaw.

Contribution

We extend a gradient-based method to show that embeddings of different texts and images can be aligned through subtle adversarial modifications, revealing a critical vulnerability.

Findings

Achieved 100% success rate in aligning texts to images across multiple datasets.

Unnoticeable adversarial attacks can misalign embeddings without perceptible changes.

Multimodal models are vulnerable to attacks that compromise their semantic alignment capabilities.

Abstract

Utilizing a shared embedding space, emerging multimodal models exhibit unprecedented zero-shot capabilities. However, the shared embedding space could lead to new vulnerabilities if different modalities can be misaligned. In this paper, we extend and utilize a recently developed effective gradient-based procedure that allows us to match the embedding of a given text by minimally modifying an image. Using the procedure, we show that we can align the embeddings of distinguishable texts to any image through unnoticeable adversarial attacks in joint image-text models, revealing that semantically unrelated images can have embeddings of identical texts and at the same time visually indistinguishable images can be matched to the embeddings of very different texts. Our technique achieves 100\% success rate when it is applied to text datasets and images from multiple sources. Without overcoming the vulnerability, multimodal models cannot robustly align inputs from different modalities in a semantically meaningful way. \textbf{Warning: the text data used in this paper are toxic in nature and may be offensive to some readers.}