BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning

TL;DR

This paper introduces BEAT, a novel framework for injecting visual backdoors into VLM-based embodied agents using object triggers, revealing significant security vulnerabilities and proposing a contrastive learning approach to enhance attack success.

Contribution

BEAT is the first method to reliably implant visual backdoors in VLM-based embodied agents with object triggers, employing a two-stage training scheme including contrastive trigger learning.

Findings

Achieves up to 80% attack success rate across benchmarks.

Boosts backdoor activation accuracy by 39% with limited data.

Maintains strong performance on benign tasks.

Abstract

Recent advances in Vision-Language Models (VLMs) have propelled embodied agents by enabling direct perception, reasoning, and planning task-oriented actions from visual inputs. However, such vision-driven embodied agents open a new attack surface: visual backdoor attacks, where the agent behaves normally until a visual trigger appears in the scene, then persistently executes an attacker-specified multi-step policy. We introduce BEAT, the first framework to inject such visual backdoors into VLM-based embodied agents using objects in the environments as triggers. Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably. BEAT addresses this challenge by (1) constructing a training set that spans diverse scenes, tasks, and trigger placements to expose agents to trigger variability, and (2) introducing a two-stage training scheme that first applies supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning (CTL). CTL formulates trigger discrimination as preference learning between trigger-present and trigger-free inputs, explicitly sharpening the decision boundaries to ensure precise backdoor activation. Across various embodied agent benchmarks and VLMs, BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance, and generalizes reliably to out-of-distribution trigger placements. Notably, compared to naive SFT, CTL boosts backdoor activation accuracy up to 39% under limited backdoor data. These findings expose a critical yet unexplored security risk in VLM-based embodied agents, underscoring the need for robust defenses before real-world deployment.