Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces

TL;DR

This paper introduces VQ-CD, a novel method for continual offline reinforcement learning that aligns different task spaces using vector quantization and selectively activates model weights, achieving state-of-the-art results across diverse tasks.

Contribution

The paper proposes a new approach combining vector quantization for space alignment with selective weight activation to handle diverse task spaces in continual offline RL.

Findings

VQ-CD outperforms 16 baselines on 15 diverse CL tasks.

Effective alignment of different state and action spaces.

Achieves state-of-the-art performance in both identical and varied space settings.

Abstract

Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based lifelong learning systems by modeling the joint distributions of trajectories. However, most research only focuses on limited continual task settings where the tasks have the same observation and action space, which deviates from the realistic demands of training agents in various environments. In view of this, we propose Vector-Quantized Continual Diffuser, named VQ-CD, to break the barrier of different spaces between various tasks. Specifically, our method contains two complementary sections, where the quantization spaces alignment provides a unified basis for the selective weights activation. In the quantized spaces alignment, we leverage vector quantization to align the different state and action spaces of various tasks, facilitating continual training in the same space. Then, we propose to leverage a unified diffusion model attached by the inverse dynamic model to master all tasks by selectively activating different weights according to the task-related sparse masks. Finally, we conduct extensive experiments on 15 continual learning (CL) tasks, including conventional CL task settings (identical state and action spaces) and general CL task settings (various state and action spaces). Compared with 16 baselines, our method reaches the SOTA performance.