Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning

TL;DR

This paper introduces a multi-level certified defense mechanism for offline reinforcement learning that provides stronger guarantees against poisoning attacks, ensuring robustness across various environments and significantly improving safety.

Contribution

The work extends certified defenses using Differential Privacy to cover both per-state actions and cumulative rewards in offline RL, applicable to continuous, discrete, stochastic, and deterministic settings.

Findings

Performance drops limited to 50% with 7% poisoned data

Achieves 5 times larger certified radii than prior work

Significantly improves robustness and safety in offline RL

Abstract

Similar to other machine learning frameworks, Offline Reinforcement Learning (RL) is shown to be vulnerable to poisoning attacks, due to its reliance on externally sourced datasets, a vulnerability that is exacerbated by its sequential nature. To mitigate the risks posed by RL poisoning, we extend certified defenses to provide larger guarantees against adversarial manipulation, ensuring robustness for both per-state actions, and the overall expected cumulative reward. Our approach leverages properties of Differential Privacy, in a manner that allows this work to span both continuous and discrete spaces, as well as stochastic and deterministic environments -- significantly expanding the scope and applicability of achievable guarantees. Empirical evaluations demonstrate that our approach ensures the performance drops to no more than $50\%$ with up to $7\%$ of the training data poisoned, significantly improving over the $0.008\%$ in prior work~\citep{wu_copa_2022}, while producing certified radii that is $5$ times larger as well. This highlights the potential of our framework to enhance safety and reliability in offline RL.