Adversarial Environment Design via Regret-Guided Diffusion Models

TL;DR

This paper introduces ADD, a novel adversarial environment design method using regret-guided diffusion models to generate challenging yet diverse training environments, improving agent robustness and generalization in reinforcement learning.

Contribution

The paper presents a new UED algorithm that leverages diffusion models guided by agent regret to produce diverse, adversarial environments for robust policy learning.

Findings

ADD outperforms baseline UED methods in zero-shot generalization

The method effectively generates challenging environments that enhance agent robustness

Diffusion models enable direct and diverse environment generation

Abstract

Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments. Project page: https://rllab-snu.github.io/projects/ADD