Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learning

TL;DR

This paper empirically investigates how plasticity interventions in deep reinforcement learning affect backdoor vulnerabilities, revealing that most interventions mitigate threats except SAM, which worsens them, and proposing new detection insights.

Contribution

It systematically studies 14,664 cases of interventions and attacks, introduces the SCC framework, and identifies loss landscape sharpness as a backdoor detection indicator.

Findings

Only SAM intervention worsens backdoor threats.

Most interventions mitigate backdoor vulnerabilities.

Loss landscape sharpness indicates backdoor presence.

Abstract

Extensive research has highlighted the severe threats posed by backdoor attacks to deep reinforcement learning (DRL). However, prior studies primarily focus on vanilla scenarios, while plasticity interventions have emerged as indispensable built-in components of modern DRL agents. Despite their effectiveness in mitigating plasticity loss, the impact of these interventions on DRL backdoor vulnerabilities remains underexplored, and this lack of systematic investigation poses risks in practical DRL deployments. To bridge this gap, we empirically study 14,664 cases integrating representative interventions and attack scenarios. We find that only one intervention (i.e., SAM) exacerbates backdoor threats, while other interventions mitigate them. Pathological analysis identifies that the exacerbation is attributed to backdoor gradient amplification, while the mitigation stems from activation pathway disruption and representation space compression. From these findings, we derive two novel insights: (1) a conceptual framework SCC for robust backdoor injection that deconstructs the mechanistic interplay between interventions and backdoors in DRL, and (2) abnormal loss landscape sharpness as a key indicator for DRL backdoor detection.