Goal-Conditioned Reinforcement Learning in the Presence of an Adversary

TL;DR

This paper introduces new environments, algorithms, and a framework for goal-conditioned reinforcement learning in adversarial settings, demonstrating improved robustness and performance against various adversaries.

Contribution

It presents novel environments, algorithms, and a unified framework for goal-conditioned RL under adversarial conditions, advancing robustness in real-world applications.

Findings

IGOAL with EHER outperforms existing methods against adversaries.

New environments DigitFlip and CLEVR-Play support adversarial scenarios.

Algorithms EHER and CHER improve goal-conditioned learning in adversarial contexts.

Abstract

Reinforcement learning has seen increasing applications in real-world contexts over the past few years. However, physical environments are often imperfect and policies that perform well in simulation might not achieve the same performance when applied elsewhere. A common approach to combat this is to train agents in the presence of an adversary. An adversary acts to destabilise the agent, which learns a more robust policy and can better handle realistic conditions. Many real-world applications of reinforcement learning also make use of goal-conditioning: this is particularly useful in the context of robotics, as it allows the agent to act differently, depending on which goal is selected. Here, we focus on the problem of goal-conditioned learning in the presence of an adversary. We first present DigitFlip and CLEVR-Play, two novel goal-conditioned environments that support acting against an adversary. Next, we propose EHER and CHER -- two HER-based algorithms for goal-conditioned learning -- and evaluate their performance. Finally, we unify the two threads and introduce IGOAL: a novel framework for goal-conditioned learning in the presence of an adversary. Experimental results show that combining IGOAL with EHER allows agents to significantly outperform existing approaches, when acting against both random and competent adversaries.