Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning

TL;DR

This paper introduces HiSPO, a hierarchical policy framework for continual offline reinforcement learning that effectively adapts to new navigation tasks while retaining prior knowledge, demonstrating efficiency and scalability in diverse environments.

Contribution

The paper presents HiSPO, a novel hierarchical policy subspace method tailored for continual offline RL in navigation tasks, addressing forgetting and scalability issues.

Findings

HiSPO outperforms baseline methods in MuJoCo maze tasks.

It maintains high performance while using less memory.

Effective in complex video game navigation simulations.

Abstract

We consider a Continual Reinforcement Learning setup, where a learning agent must continuously adapt to new tasks while retaining previously acquired skill sets, with a focus on the challenge of avoiding forgetting past gathered knowledge and ensuring scalability with the growing number of tasks. Such issues prevail in autonomous robotics and video game simulations, notably for navigation tasks prone to topological or kinematic changes. To address these issues, we introduce HiSPO, a novel hierarchical framework designed specifically for continual learning in navigation settings from offline data. Our method leverages distinct policy subspaces of neural networks to enable flexible and efficient adaptation to new tasks while preserving existing knowledge. We demonstrate, through a careful experimental study, the effectiveness of our method in both classical MuJoCo maze environments and complex video game-like navigation simulations, showcasing competitive performances and satisfying adaptability with respect to classical continual learning metrics, in particular regarding the memory usage and efficiency.