Autonomous Curriculum Design via Relative Entropy Based Task Modifications

TL;DR

This paper introduces an autonomous curriculum design method that uses relative entropy to select high-uncertainty tasks, improving learning efficiency without human intervention.

Contribution

It presents a novel uncertainty-based curriculum design algorithm with theoretical convergence guarantees, outperforming existing methods and supporting both autonomous and teacher-guided learning.

Findings

Outperforms random curriculum and direct target task learning

Provides theoretical guarantees for convergence

Supports hybrid autonomous and teacher-guided curriculum design

Abstract

Curriculum learning is a training method in which an agent is first trained on a curriculum of relatively simple tasks related to a target task in an effort to shorten the time required to train on the target task. Autonomous curriculum design involves the design of such curriculum with no reliance on human knowledge and/or expertise. Finding an efficient and effective way of autonomously designing curricula remains an open problem. We propose a novel approach for automatically designing curricula by leveraging the learner's uncertainty to select curricula tasks. Our approach measures the uncertainty in the learner's policy using relative entropy, and guides the agent to states of high uncertainty to facilitate learning. Our algorithm supports the generation of autonomous curricula in a self-assessed manner by leveraging the learner's past and current policies but it also allows the use of teacher guided design in an instructive setting. We provide theoretical guarantees for the convergence of our algorithm using two time-scale optimization processes. Results show that our algorithm outperforms randomly generated curriculum, and learning directly on the target task as well as the curriculum-learning criteria existing in literature. We also present two additional heuristic distance measures that could be combined with our relative-entropy approach for further performance improvements.