Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning

TL;DR

This paper introduces Qreg+NWLU, a novel data rehearsal method for Deep Q-Networks in multi-cyclic continual reinforcement learning, improving learning efficiency and reducing forgetting.

Contribution

It extends data rehearsal to value function approximation in multi-cyclic CRL and proposes two simple modifications that enhance performance.

Findings

Qreg+NWLU outperforms existing methods in multi-cyclic environments.

Continuous data rehearsal improves knowledge retention.

Immediate regularization enhances learning efficiency.

Abstract

Data rehearsal has emerged as a leading approach for mitigating catastrophic forgetting in Continual Reinforcement Learning (CRL). However, existing work remains confined to policy gradient frameworks, regularizing only actors due to the performance degradation incurred by critic regularization. This actor-centric approach overlooks the potential of data rehearsal for value function approximation. Moreover, existing evaluations in CRL rarely consider multi-cyclic environments where task sequences repeat, a critical real-world scenario that exacerbates forgetting and plasticity. We investigate data rehearsal for Deep Q-Networks using Q-value regularization in multi-cyclic settings and propose Qreg+NWLU which introduces two simple modifications: (1) continuous data rehearsal that dynamically collects and updates stored Q-values throughout training, and (2) "No-Wait" regularization that applies immediately rather than after the first task. Together, these modifications yield improvements in learning efficiency, forgetting mitigation, and knowledge transfer over Qreg and conventional CRL methods within value function approximation settings.