Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal

TL;DR

This paper introduces Continual Diffuser (CoD), a rehearsal-based diffusion model designed for continual offline reinforcement learning, capable of adapting to new tasks while retaining previously learned knowledge, demonstrated on a diverse multi-task benchmark.

Contribution

The paper presents a novel continual diffusion model with rehearsal strategies for reinforcement learning, addressing the plasticity-stability trade-off in sequential task learning.

Findings

CoD outperforms existing diffusion-based methods on most tasks.

The model effectively balances adaptation and knowledge retention.

Extensive experiments validate its superiority in multi-task settings.

Abstract

Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks.