Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments

TL;DR

This paper introduces demonstration-guided continual reinforcement learning (DGCRL), which uses an external demonstration repository to improve learning efficiency, knowledge transfer, and stability in dynamic environments by guiding exploration and adaptation.

Contribution

The paper proposes a novel DGCRL framework that leverages a self-evolving demonstration repository to directly guide RL agents, enhancing continual learning in changing environments.

Findings

DGCRL outperforms existing methods in 2D navigation and MuJoCo tasks.

It improves knowledge transfer and reduces forgetting.

Training efficiency is significantly enhanced.

Abstract

Reinforcement learning (RL) excels in various applications but struggles in dynamic environments where the underlying Markov decision process evolves. Continual reinforcement learning (CRL) enables RL agents to continually learn and adapt to new tasks, but balancing stability (preserving prior knowledge) and plasticity (acquiring new knowledge) remains challenging. Existing methods primarily address the stability-plasticity dilemma through mechanisms where past knowledge influences optimization but rarely affects the agent's behavior directly, which may hinder effective knowledge reuse and efficient learning. In contrast, we propose demonstration-guided continual reinforcement learning (DGCRL), which stores prior knowledge in an external, self-evolving demonstration repository that directly guides RL exploration and adaptation. For each task, the agent dynamically selects the most relevant demonstration and follows a curriculum-based strategy to accelerate learning, gradually shifting from demonstration-guided exploration to fully self-exploration. Extensive experiments on 2D navigation and MuJoCo locomotion tasks demonstrate its superior average performance, enhanced knowledge transfer, mitigation of forgetting, and training efficiency. The additional sensitivity analysis and ablation study further validate its effectiveness.