Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation

TL;DR

This paper introduces an off-policy reinforcement learning method that enhances robustness against adversarial observations by reformulating the problem as a soft-constrained optimization, supported by symmetric policy evaluation theory.

Contribution

It presents a novel off-policy approach that removes the need for extra environment interactions in adversarial RL, leveraging symmetric policy evaluation for robustness.

Findings

Eliminates additional environment interactions in adversarial training

Uses symmetric policy evaluation to support the approach

Demonstrates improved robustness in RL agents

Abstract

Recently, robust reinforcement learning (RL) methods designed to handle adversarial input observations have received significant attention, motivated by RL's inherent vulnerabilities. While existing approaches have demonstrated reasonable success, addressing worst-case scenarios over long time horizons requires both minimizing the agent's cumulative rewards for adversaries and training agents to counteract them through alternating learning. However, this process introduces mutual dependencies between the agent and the adversary, making interactions with the environment inefficient and hindering the development of off-policy methods. In this work, we propose a novel off-policy method that eliminates the need for additional environmental interactions by reformulating adversarial learning as a soft-constrained optimization problem. Our approach is theoretically supported by the symmetric property of policy evaluation between the agent and the adversary. The implementation is available at https://github.com/nakanakakosuke/VALT_SAC.