Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses

TL;DR

This paper proposes a framework that improves the robustness of offline reinforcement learning models by applying adversarial attacks and defenses, demonstrating increased resilience against observation perturbations on benchmark datasets.

Contribution

It introduces a novel framework combining multiple adversarial attacks and defenses to enhance offline RL policy robustness, which is less explored in prior work.

Findings

Attacks significantly degrade policy performance.

Defenses effectively improve robustness.

Framework enhances reliability in practical scenarios.

Abstract

Offline reinforcement learning (RL) addresses the challenge of expensive and high-risk data exploration inherent in RL by pre-training policies on vast amounts of offline data, enabling direct deployment or fine-tuning in real-world environments. However, this training paradigm can compromise policy robustness, leading to degraded performance in practical conditions due to observation perturbations or intentional attacks. While adversarial attacks and defenses have been extensively studied in deep learning, their application in offline RL is limited. This paper proposes a framework to enhance the robustness of offline RL models by leveraging advanced adversarial attacks and defenses. The framework attacks the actor and critic components by perturbing observations during training and using adversarial defenses as regularization to enhance the learned policy. Four attacks and two defenses are introduced and evaluated on the D4RL benchmark. The results show the vulnerability of both the actor and critic to attacks and the effectiveness of the defenses in improving policy robustness. This framework holds promise for enhancing the reliability of offline RL models in practical scenarios.