Dissecting Adversarial Robustness of Multimodal LM Agents

TL;DR

This paper introduces the ARE framework to evaluate the adversarial robustness of multimodal language model agents, revealing vulnerabilities in recent systems through targeted attacks and analyzing how component additions affect robustness.

Contribution

We propose the Agent Robustness Evaluation (ARE) framework for systematic robustness assessment of multimodal LM agents, including a new threat model and attack methods.

Findings

Latest agents are vulnerable to imperceptible image perturbations with up to 67% success rate.

Adding new components can decrease robustness, with attack success increasing by up to 20%.

Inference-time compute may unintentionally introduce new vulnerabilities.

Abstract

As language models (LMs) are used to build autonomous agents in real environments, ensuring their adversarial robustness becomes a critical challenge. Unlike chatbots, agents are compound systems with multiple components taking actions, which existing LMs safety evaluations do not adequately address. To bridge this gap, we manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena, a real environment for web agents. To systematically examine the robustness of agents, we propose the Agent Robustness Evaluation (ARE) framework. ARE views the agent as a graph showing the flow of intermediate outputs between components and decomposes robustness as the flow of adversarial information on the graph. We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search. With imperceptible perturbations to a single image (less than 5% of total web page pixels), an attacker can hijack these agents to execute targeted adversarial goals with success rates up to 67%. We also use ARE to rigorously evaluate how the robustness changes as new components are added. We find that inference-time compute that typically improves benign performance can open up new vulnerabilities and harm robustness. An attacker can compromise the evaluator used by the reflexion agent and the value function of the tree search agent, which increases the attack success relatively by 15% and 20%. Our data and code for attacks, defenses, and evaluation are at https://github.com/ChenWu98/agent-attack