From Prompt to Physical Action: Structured Backdoor Attacks on LLM-Mediated Robotic Control Systems

TL;DR

This paper uncovers vulnerabilities in LLM-based robotic control systems, demonstrating how structured backdoors can reliably trigger physical actions with high success rates and proposing a defense mechanism using semantic verification.

Contribution

It introduces supply-chain backdoor strategies in LLM-mediated robotic control, revealing their effectiveness and exploring a semantic verification defense to mitigate attacks.

Findings

Backdoors at command format levels are more effective than natural-language reasoning backdoors.

Backdoored models achieve 83% attack success rate with over 93% accuracy and low latency.

Semantic verification reduces attack success to 20% but increases response latency.

Abstract

The integration of large language models (LLMs) into robotic control pipelines enables natural language interfaces that translate user prompts into executable commands. However, this digital-to-physical interface introduces a critical and underexplored vulnerability: structured backdoor attacks embedded during fine-tuning. In this work, we experimentally investigate LoRA-based supply-chain backdoors in LLM-mediated ROS2 robotic control systems and evaluate their impact on physical robot execution. We construct two poisoned fine-tuning strategies targeting different stages of the command generation pipeline and reveal a key systems-level insight: back-doors embedded at the natural-language reasoning stage do not reliably propagate to executable control outputs, whereas backdoors aligned directly with structured JSON command formats successfully survive translation and trigger physical actions. In both simulation and real-world experiments, backdoored models achieve an average Attack Success Rate of 83% while maintaining over 93% Clean Performance Accuracy (CPA) and sub-second latency, demonstrating both reliability and stealth. We further implement an agentic verification defense using a secondary LLM for semantic consistency checking. Although this reduces the Attack Success Rate (ASR) to 20%, it increases end-to-end latency to 8-9 seconds, exposing a significant security-responsiveness trade-off in real-time robotic systems. These results highlight structural vulnerabilities in LLM-mediated robotic control architectures and underscore the need for robotics-aware defenses for embodied AI systems.