Proximal Curriculum for Reinforcement Learning Agents

TL;DR

This paper introduces ProCuRL, a curriculum strategy inspired by ZPD, that dynamically selects tasks to optimize learning progress in reinforcement learning agents, demonstrating improved training efficiency across various domains.

Contribution

We propose ProCuRL, a theoretically grounded curriculum method inspired by ZPD, with a practical variant for deep RL that outperforms existing approaches.

Findings

ProCuRL accelerates training of deep RL agents.

ProCuRL outperforms state-of-the-art curriculum methods.

Theoretical analysis supports ProCuRL's effectiveness.

Abstract

We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task settings. Existing techniques on automatic curriculum design typically require domain-specific hyperparameter tuning or have limited theoretical underpinnings. To tackle these limitations, we design our curriculum strategy, ProCuRL, inspired by the pedagogical concept of Zone of Proximal Development (ZPD). ProCuRL captures the intuition that learning progress is maximized when picking tasks that are neither too hard nor too easy for the learner. We mathematically derive ProCuRL by analyzing two simple learning settings. We also present a practical variant of ProCuRL that can be directly integrated with deep RL frameworks with minimal hyperparameter tuning. Experimental results on a variety of domains demonstrate the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.