Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers

TL;DR

This paper exposes a vulnerability in vision-language models used in search rankings, showing how adversaries can manipulate rankings through combined image and text perturbations, highlighting a significant security concern.

Contribution

The paper introduces MGEO, a novel adversarial framework that jointly optimizes image and text perturbations to manipulate multimodal ranking systems, revealing a new security threat.

Findings

Multimodal attacks outperform unimodal baselines.

Adversarial perturbations can be imperceptible yet effective.

VLMs' cross-modal coupling can be exploited for manipulation.

Abstract

Vision-Language Models (VLMs) are rapidly replacing unimodal encoders in modern retrieval and recommendation systems. While their capabilities are well-documented, their robustness against adversarial manipulation in competitive ranking scenarios remains largely unexplored. In this paper, we uncover a critical vulnerability in VLM-based product search: multimodal ranking attacks. We present Multimodal Generative Engine Optimization (MGEO), a novel adversarial framework that enables a malicious actor to unfairly promote a target product by jointly optimizing imperceptible image perturbations and fluent textual suffixes. Unlike existing attacks that treat modalities in isolation, MGEO employs an alternating gradient-based optimization strategy to exploit the deep cross-modal coupling within the VLM. Extensive experiments on real-world datasets using state-of-the-art models demonstrate that our coordinated attack significantly outperforms text-only and image-only baselines. These findings reveal that multimodal synergy, typically a strength of VLMs, can be weaponized to compromise the integrity of search rankings without triggering conventional content filters.