Faster Reinforcement Learning Using Active Simulators

TL;DR

This paper introduces online methods to construct effective training curricula from multiple tasks, significantly accelerating reinforcement learning by optimizing task sequences and leveraging domain knowledge.

Contribution

It presents novel online curriculum construction methods that improve RL training efficiency and are adaptable across different algorithms and domains.

Findings

Methods reduce total training time compared to training on target task alone.

Curricula tailored with domain knowledge further enhance learning speed.

Applicable across multiple domains and RL algorithms.

Abstract

In this work, we propose several online methods to build a \emph{learning curriculum} from a given set of target-task-specific training tasks in order to speed up reinforcement learning (RL). These methods can decrease the total training time needed by an RL agent compared to training on the target task from scratch. Unlike traditional transfer learning, we consider creating a sequence from several training tasks in order to provide the most benefit in terms of reducing the total time to train. Our methods utilize the learning trajectory of the agent on the curriculum tasks seen so far to decide which tasks to train on next. An attractive feature of our methods is that they are weakly coupled to the choice of the RL algorithm as well as the transfer learning method. Further, when there is domain information available, our methods can incorporate such knowledge to further speed up the learning. We experimentally show that these methods can be used to obtain suitable learning curricula that speed up the overall training time on two different domains.