Parallelized Reverse Curriculum Generation

TL;DR

This paper introduces a parallelized reverse curriculum generation method for reinforcement learning that trains multiple actor-critic pairs simultaneously, improving performance and convergence in sparse reward tasks.

Contribution

It proposes a novel parallelized approach to reverse curriculum generation that enhances generalization and speeds up training in actor-critic reinforcement learning algorithms.

Findings

Improved performance in RL tasks with sparse rewards.

Faster convergence compared to traditional RCG methods.

Applicable to various actor-critic algorithms.

Abstract

For reinforcement learning (RL), it is challenging for an agent to master a task that requires a specific series of actions due to sparse rewards. To solve this problem, reverse curriculum generation (RCG) provides a reverse expansion approach that automatically generates a curriculum for the agent to learn. More specifically, RCG adapts the initial state distribution from the neighborhood of a goal to a distance as training proceeds. However, the initial state distribution generated for each iteration might be biased, thus making the policy overfit or slowing down the reverse expansion rate. While training RCG for actor-critic (AC) based RL algorithms, this poor generalization and slow convergence might be induced by the tight coupling between an AC pair. Therefore, we propose a parallelized approach that simultaneously trains multiple AC pairs and periodically exchanges their critics. We empirically demonstrate that this proposed approach can improve RCG in performance and convergence, and it can also be applied to other AC based RL algorithms with adapted initial state distribution.