Understanding the Complexity Gains of Single-Task RL with a Curriculum

TL;DR

This paper demonstrates that reformulating single-task reinforcement learning as a multi-task problem with a curriculum can lead to more efficient learning, both theoretically and practically, especially in robotic tasks.

Contribution

The paper provides a theoretical framework showing the efficiency of curriculum-based multi-task RL for single-task problems and introduces a practical algorithm for robotic applications.

Findings

Sequential task solving is more computationally efficient than single-task RL.

Theoretical conditions ensure efficiency gains with curriculum reformulation.

Practical algorithm accelerates learning in robotic simulations.

Abstract

Reinforcement learning (RL) problems can be challenging without well-shaped rewards. Prior work on provably efficient RL methods generally proposes to address this issue with dedicated exploration strategies. However, another way to tackle this challenge is to reformulate it as a multi-task RL problem, where the task space contains not only the challenging task of interest but also easier tasks that implicitly function as a curriculum. Such a reformulation opens up the possibility of running existing multi-task RL methods as a more efficient alternative to solving a single challenging task from scratch. In this work, we provide a theoretical framework that reformulates a single-task RL problem as a multi-task RL problem defined by a curriculum. Under mild regularity conditions on the curriculum, we show that sequentially solving each task in the multi-task RL problem is more computationally efficient than solving the original single-task problem, without any explicit exploration bonuses or other exploration strategies. We also show that our theoretical insights can be translated into an effective practical learning algorithm that can accelerate curriculum learning on simulated robotic tasks.