Policy Resilience to Environment Poisoning Attacks on Reinforcement Learning

TL;DR

This paper presents a resource-efficient, knowledge-sharing-based policy resilience mechanism for reinforcement learning that quickly diagnoses and recovers from environment poisoning attacks, maintaining policy performance.

Contribution

It introduces a federated, meta-learning approach for policy resilience, enabling rapid detection and recovery from poisoning attacks in RL policies.

Findings

Effective in restoring policy performance after poisoning

Applicable to both model-based and model-free RL algorithms

Demonstrates efficiency and robustness in empirical evaluations

Abstract

This paper investigates policy resilience to training-environment poisoning attacks on reinforcement learning (RL) policies, with the goal of recovering the deployment performance of a poisoned RL policy. Due to the fact that the policy resilience is an add-on concern to RL algorithms, it should be resource-efficient, time-conserving, and widely applicable without compromising the performance of RL algorithms. This paper proposes such a policy-resilience mechanism based on an idea of knowledge sharing. We summarize the policy resilience as three stages: preparation, diagnosis, recovery. Specifically, we design the mechanism as a federated architecture coupled with a meta-learning manner, pursuing an efficient extraction and sharing of the environment knowledge. With the shared knowledge, a poisoned agent can quickly identify the deployment condition and accordingly recover its policy performance. We empirically evaluate the resilience mechanism for both model-based and model-free RL algorithms, showing its effectiveness and efficiency in restoring the deployment performance of a poisoned policy.