Reward-Preserving Attacks For Robust Reinforcement Learning

TL;DR

This paper introduces reward-preserving adversarial attacks in reinforcement learning that adapt perturbation strength dynamically to maintain a specified return gap, leading to more robust policies.

Contribution

It proposes a novel adaptive attack method using a learned critic to preserve reward levels, improving robustness over fixed or random perturbation strategies.

Findings

Adaptive attacks outperform fixed-radius methods.

Policies trained with adaptive attacks are robust across various perturbation magnitudes.

The method maintains nominal performance while enhancing robustness.

Abstract

Adversarial training in reinforcement learning (RL) is challenging because perturbations cascade through trajectories and compound over time, making fixed-strength attacks either overly destructive or too conservative. We propose reward-preserving attacks, which adapt adversarial strength so that an $\alpha$ fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, perturbation magnitudes $\eta$ are selected dynamically, using a learned critic $Q((s,a),\eta)$ that estimates the expected return of $\alpha$-reward-preserving rollouts. For intermediate values of $\alpha$, this adaptive training yields policies that are robust across a wide range of perturbation magnitudes while preserving nominal performance, outperforming fixed-radius and uniformly sampled-radius adversarial training.