Continual Policy Distillation from Distributed Reinforcement Learning Teachers

TL;DR

This paper introduces a teacher-student framework for continual reinforcement learning that distills distributed single-task teachers into a generalist model, improving scalability and reducing forgetting across tasks.

Contribution

It proposes a novel decoupled approach combining distributed RL teachers with policy distillation and MoE architecture for scalable continual RL.

Findings

Achieves over 85% of teacher performance on Meta-World.

Limits task-wise forgetting to within 10%.

Enhances stability and plasticity in continual policy learning.

Abstract

Continual Reinforcement Learning (CRL) aims to develop lifelong learning agents to continuously acquire knowledge across diverse tasks while mitigating catastrophic forgetting. This requires efficiently managing the stability-plasticity dilemma and leveraging prior experience to rapidly generalize to novel tasks. While various enhancement strategies for both aspects have been proposed, achieving scalable performance by directly applying RL to sequential task streams remains challenging. In this paper, we propose a novel teacher-student framework that decouples CRL into two independent processes: training single-task teacher models through distributed RL and continually distilling them into a central generalist model. This design is motivated by the observation that RL excels at solving single tasks, while policy distillation -- a relatively stable supervised learning process -- is well aligned with large foundation models and multi-task learning. Moreover, a mixture-of-experts (MoE) architecture and a replay-based approach are employed to enhance the plasticity and stability of the continual policy distillation process. Extensive experiments on the Meta-World benchmark demonstrate that our framework enables efficient continual RL, recovering over 85% of teacher performance while constraining task-wise forgetting to within 10%.