DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning

TL;DR

This paper introduces DCUR, a data curriculum framework for reinforcement learning that improves sample efficiency and performance by strategically limiting and expanding data access during training.

Contribution

The paper proposes a novel data curriculum approach for RL, demonstrating how to effectively transfer teacher policies to students using staged data exposure.

Findings

Data curricula significantly influence student learning outcomes.

Limiting data early and expanding over time enhances training efficiency.

Small amounts of online data complement the data curriculum benefits.

Abstract

Deep reinforcement learning (RL) has shown great empirical successes, but suffers from brittleness and sample inefficiency. A potential remedy is to use a previously-trained policy as a source of supervision. In this work, we refer to these policies as teachers and study how to transfer their expertise to new student policies by focusing on data usage. We propose a framework, Data CUrriculum for Reinforcement learning (DCUR), which first trains teachers using online deep RL, and stores the logged environment interaction history. Then, students learn by running either offline RL or by using teacher data in combination with a small amount of self-generated data. DCUR's central idea involves defining a class of data curricula which, as a function of training time, limits the student to sampling from a fixed subset of the full teacher data. We test teachers and students using state-of-the-art deep RL algorithms across a variety of data curricula. Results suggest that the choice of data curricula significantly impacts student learning, and that it is beneficial to limit the data during early training stages while gradually letting the data availability grow over time. We identify when the student can learn offline and match teacher performance without relying on specialized offline RL algorithms. Furthermore, we show that collecting a small fraction of online data provides complementary benefits with the data curriculum. Supplementary material is available at https://tinyurl.com/teach-dcur.