Semantic Router: On the Feasibility of Hijacking MLLMs via a Single Adversarial Perturbation

TL;DR

This paper demonstrates the feasibility of hijacking multimodal large language models using a single universal perturbation that perceives semantics and routes inputs to attacker-defined targets.

Contribution

It introduces the Semantic-Aware Universal Perturbation (SAUP) and the Semantic-Oriented (SORT) optimization strategy, providing a novel method for semantic hijacking of MLLMs.

Findings

Achieved 66% attack success rate over five targets on Qwen.

Validated the attack's effectiveness across three different MLLMs.

Provided theoretical and empirical analysis of latent space geometry.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed in stateless systems, such as autonomous driving and robotics. This paper investigates a novel threat: Semantic-Aware Hijacking. We explore the feasibility of hijacking multiple stateless decisions simultaneously using a single universal perturbation. We introduce the Semantic-Aware Universal Perturbation (SAUP), which acts as a semantic router, "actively" perceiving input semantics and routing them to distinct, attacker-defined targets. To achieve this, we conduct theoretical and empirical analysis on the geometric properties in the latent space. Guided by these insights, we propose the Semantic-Oriented (SORT) optimization strategy and annotate a new dataset with fine-grained semantics to evaluate performance. Extensive experiments on three representative MLLMs demonstrate the fundamental feasibility of this attack, achieving a 66% attack success rate over five targets using a single frame against Qwen.