ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior

TL;DR

This paper introduces ARGUS, a novel modality-independent defense mechanism that steers MLLMs' representations to resist multimodal prompt injections while maintaining utility, through optimal subspace search and adaptive steering.

Contribution

The paper proposes ARGUS, a new method that effectively defends against multimodal IPI attacks by steering model representations within a safety subspace, balancing security and utility.

Findings

ARGUS achieves robust defense against multimodal IPI attacks.

It preserves model utility better than naive steering methods.

The approach includes lightweight detection and post-filtering for on-demand activation.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly vulnerable to multimodal Indirect Prompt Injection (IPI) attacks, which embed malicious instructions in images, videos, or audio to hijack model behavior. Existing defenses, designed primarily for text-only LLMs, are unsuitable for countering these multimodal threats, as they are easily bypassed, modality-dependent, or generalize poorly. Inspired by activation steering researches, we hypothesize that a robust, general defense independent of modality can be achieved by steering the model's behavior in the representation space. Through extensive experiments, we discover that the instruction-following behavior of MLLMs is encoded in a subspace. Steering along directions within this subspace can enforce adherence to user instructions, forming the basis of a defense. However, we also found that a naive defense direction could be coupled with a utility-degrading direction, and excessive intervention strength harms model performance. To address this, we propose ARGUS, which searches for an optimal defense direction within the safety subspace that decouples from the utility degradation direction, further combining adaptive strength steering to achieve a better safety-utility trade-off. ARGUS also introduces lightweight injection detection stage to activate the defense on-demand, and a post-filtering stage to verify defense success. Experimental results show that ARGUS can achieve robust defense against multimodal IPI while maximally preserving the MLLM's utility.