Syllabus: Portable Curricula for Reinforcement Learning Agents

TL;DR

Syllabus is a portable, easy-to-integrate library that standardizes curriculum learning in reinforcement learning, enabling rapid development and testing across diverse environments and RL frameworks.

Contribution

It introduces Syllabus, a universal API and infrastructure for curriculum learning, simplifying integration and fostering innovation in complex RL environments.

Findings

Demonstrated in NetHack and Neural MMO environments

Existing curriculum methods do not transfer well to complex tasks

Syllabus facilitates rapid prototyping and testing of curriculum algorithms

Abstract

Curriculum learning has been a quiet, yet crucial component of many high-profile successes of reinforcement learning. Despite this, it is still a niche topic that is not directly supported by any of the major reinforcement learning libraries. These methods can improve the capabilities and generalization of RL agents, but often require complex changes to training code. We introduce Syllabus, a portable curriculum learning library, as a solution to this problem. Syllabus provides a universal API for curriculum learning, modular implementations of popular automatic curriculum learning methods, and infrastructure that allows them to be easily integrated with asynchronous training code in nearly any RL library. Syllabus provides a minimal API for core curriculum learning components, making it easier to design new algorithms and adapt existing ones to new environments. We demonstrate this by evaluating the algorithms in Syllabus on several new environments, each using agents written in a different RL library. We present the first examples of automatic curriculum learning in NetHack and Neural MMO, two of the most challenging RL benchmarks, and find evidence that existing methods do not directly transfer to complex new environments. Syllabus can be found at https://github.com/RyanNavillus/Syllabus.