On Minimizing Adversarial Counterfactual Error in Adversarial RL

TL;DR

This paper introduces a novel objective called Adversarial Counterfactual Error (ACoE) to improve the robustness of Deep Reinforcement Learning policies against adversarial observation noise, addressing limitations of existing methods.

Contribution

The paper proposes ACoE and its scalable surrogate C-ACoE, a new approach that directly accounts for partial observability to enhance adversarial robustness in DRL.

Findings

Outperforms state-of-the-art adversarial RL methods on benchmarks

Effectively balances robustness and performance in benign settings

Demonstrates significant improvements in MuJoCo, Atari, and Highway environments

Abstract

Deep Reinforcement Learning (DRL) policies are highly susceptible to adversarial noise in observations, which poses significant risks in safety-critical scenarios. The challenge inherent to adversarial perturbations is that by altering the information observed by the agent, the state becomes only partially observable. Existing approaches address this by either enforcing consistent actions across nearby states or maximizing the worst-case value within adversarially perturbed observations. However, the former suffers from performance degradation when attacks succeed, while the latter tends to be overly conservative, leading to suboptimal performance in benign settings. We hypothesize that these limitations stem from their failing to account for partial observability directly. To this end, we introduce a novel objective called Adversarial Counterfactual Error (ACoE), defined on the beliefs about the true state and balancing value optimization with robustness. To make ACoE scalable in model-free settings, we propose the theoretically-grounded surrogate objective Cumulative-ACoE (C-ACoE). Our empirical evaluations on standard benchmarks (MuJoCo, Atari, and Highway) demonstrate that our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges, offering a promising direction for improving robustness in DRL under adversarial conditions. Our code is available at https://github.com/romanbelaire/acoe-robust-rl.