Online Optimization of Curriculum Learning Schedules using Evolutionary Optimization

TL;DR

This paper introduces RHEA CL, an evolutionary algorithm-based method for automatically optimizing curriculum learning schedules in reinforcement learning, showing improved early-stage performance and stable results in Minigrid environments.

Contribution

The paper presents RHEA CL, a novel approach combining curriculum learning with rolling horizon evolutionary algorithms for automatic curriculum optimization in RL.

Findings

Outperforms other curriculum learners in early training stages.

Achieves stable and superior final performance in tested environments.

Requires additional evaluation during training, increasing computational cost.

Abstract

We propose RHEA CL, which combines Curriculum Learning (CL) with Rolling Horizon Evolutionary Algorithms (RHEA) to automatically produce effective curricula during the training of a reinforcement learning agent. RHEA CL optimizes a population of curricula, using an evolutionary algorithm, and selects the best-performing curriculum as the starting point for the next training epoch. Performance evaluations are conducted after every curriculum step in all environments. We evaluate the algorithm on the \textit{DoorKey} and \textit{DynamicObstacles} environments within the Minigrid framework. It demonstrates adaptability and consistent improvement, particularly in the early stages, while reaching a stable performance later that is capable of outperforming other curriculum learners. In comparison to other curriculum schedules, RHEA CL has been shown to yield performance improvements for the final Reinforcement learning (RL) agent at the cost of additional evaluation during training.