Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning

TL;DR

This paper introduces the SCAB backdoor attack for reinforcement learning, demonstrating that limited access during training can cause significant performance degradation, highlighting security risks in untrusted RL supply chains.

Contribution

The paper proposes a novel supply-chain backdoor attack for RL that requires only legitimate interactions, not access to policy parameters, and shows its effectiveness with minimal poisoning.

Findings

Over 90% of triggered actions activated with 3% training experience poisoning

Average episodic return reduced by 80% due to the attack

Attack demonstrates risks in untrusted RL training supply-chains

Abstract

The current state-of-the-art backdoor attacks against Reinforcement Learning (RL) rely upon unrealistically permissive access models, that assume the attacker can read (or even write) the victim's policy parameters, observations, or rewards. In this work, we question whether such a strong assumption is required to launch backdoor attacks against RL. To answer this question, we propose the \underline{S}upply-\underline{C}h\underline{a}in \underline{B}ackdoor (SCAB) attack, which targets a common RL workflow: training agents using external agents that are provided separately or embedded within the environment. In contrast to prior works, our attack only relies on legitimate interactions of the RL agent with the supplied agents. Despite this limited access model, by poisoning a mere $3\%$ of training experiences, our attack can successfully activate over $90\%$ of triggered actions, reducing the average episodic return by $80\%$ for the victim. Our novel attack demonstrates that RL attacks are likely to become a reality under untrusted RL training supply-chains.