Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM

TL;DR

This technical report presents a top-ranked attack method for MLLMs using suffix injection and PGD to fool the LLaVA 1.5 model, demonstrating vulnerabilities in multimodal models.

Contribution

The paper introduces a novel combination of suffix injection and PGD for effective adversarial attacks on MLLMs, advancing attack strategies in multimodal AI.

Findings

Suffix injection successfully misleads MLLMs.

PGD adds imperceptible perturbations to images.

Combined approach achieves high attack success rate.

Abstract

This technical report introduces our top-ranked solution that employs two approaches, \ie suffix injection and projected gradient descent (PGD) , to address the TiFA workshop MLLM attack challenge. Specifically, we first append the text from an incorrectly labeled option (pseudo-labeled) to the original query as a suffix. Using this modified query, our second approach applies the PGD method to add imperceptible perturbations to the image. Combining these two techniques enables successful attacks on the LLaVA 1.5 model.