A Dual-Agent Adversarial Framework for Robust Generalization in Deep Reinforcement Learning

TL;DR

This paper introduces a dual-agent adversarial framework in deep reinforcement learning that enhances generalization by encouraging agents to learn robust policies against irrelevant high-dimensional features, demonstrated on Procgen benchmarks.

Contribution

The paper proposes a novel dual-agent adversarial learning framework that improves RL generalization without human priors, applicable across various algorithms like PPO.

Findings

Significant performance improvements on Procgen benchmark

Enhanced robustness against irrelevant features in high-dimensional observations

Applicable to multiple RL algorithms, notably PPO

Abstract

Recently, empowered with the powerful capabilities of neural networks, reinforcement learning (RL) has successfully tackled numerous challenging tasks. However, while these models demonstrate enhanced decision-making abilities, they are increasingly prone to overfitting. For instance, a trained RL model often fails to generalize to even minor variations of the same task, such as a change in background color or other minor semantic differences. To address this issue, we propose a dual-agent adversarial policy learning framework, which allows agents to spontaneously learn the underlying semantics without introducing any human prior knowledge. Specifically, our framework involves a game process between two agents: each agent seeks to maximize the impact of perturbing on the opponent's policy by producing representation differences for the same state, while maintaining its own stability against such perturbations. This interaction encourages agents to learn generalizable policies, capable of handling irrelevant features from the high-dimensional observations. Extensive experimental results on the Procgen benchmark demonstrate that the adversarial process significantly improves the generalization performance of both agents, while also being applied to various RL algorithms, e.g., Proximal Policy Optimization (PPO). With the adversarial framework, the RL agent outperforms the baseline methods by a significant margin, especially in hard-level tasks, marking a significant step forward in the generalization capabilities of deep reinforcement learning.