Robust Reinforcement Learning via Genetic Curriculum

TL;DR

This paper introduces a genetic curriculum algorithm that automatically generates training scenarios for reinforcement learning agents, significantly improving their robustness without expert supervision by adapting to the agent's performance.

Contribution

The paper presents a novel genetic curriculum method that automatically identifies failure scenarios and generates curricula, enhancing robustness in deep RL without requiring expert-designed environment adjustments.

Findings

Agents trained with our method are 2-8x less likely to fail.

Our approach outperforms existing state-of-the-art algorithms in robustness.

The method reduces the need for expert supervision in curriculum generation.

Abstract

Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While other approaches involve automatically adjusting environment setups during training, they have been limited to simple environments where low-dimensional encodings can be used. Inspired by these approaches, we propose genetic curriculum, an algorithm that automatically identifies scenarios in which the agent currently fails and generates an associated curriculum to help the agent learn to solve the scenarios and acquire more robust behaviors. As a non-parametric optimizer, our approach uses a raw, non-fixed encoding of scenarios, reducing the need for expert supervision and allowing our algorithm to adapt to the changing performance of the agent. Our empirical studies show improvement in robustness over the existing state of the art algorithms, providing training curricula that result in agents being 2 - 8x times less likely to fail without sacrificing cumulative reward. We include an ablation study and share insights on why our algorithm outperforms prior approaches.