Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning

TL;DR

This paper introduces a dual-learner framework inspired by human memory systems for continual reinforcement learning, combining a fast learner and a meta learner to improve knowledge transfer and prevent forgetting.

Contribution

The study proposes a novel dual-learner architecture with an adaptive warm-up mechanism for continual RL, explicitly addressing catastrophic forgetting and enabling efficient knowledge integration.

Findings

Outperforms baseline methods in pixel-based and continuous control benchmarks.

Effectively mitigates catastrophic forgetting during continual learning.

Enhances rapid adaptation to new environments.

Abstract

Inspired by the human learning and memory system, particularly the interplay between the hippocampus and cerebral cortex, this study proposes a dual-learner framework comprising a fast learner and a meta learner to address continual Reinforcement Learning~(RL) problems. These two learners are coupled to perform distinct yet complementary roles: the fast learner focuses on knowledge transfer, while the meta learner ensures knowledge integration. In contrast to traditional multi-task RL approaches that share knowledge through average return maximization, our meta learner incrementally integrates new experiences by explicitly minimizing catastrophic forgetting, thereby supporting efficient cumulative knowledge transfer for the fast learner. To facilitate rapid adaptation in new environments, we introduce an adaptive meta warm-up mechanism that selectively harnesses past knowledge. We conduct experiments in various pixel-based and continuous control benchmarks, revealing the superior performance of continual learning for our proposed dual-learner approach relative to baseline methods. The code is released in https://github.com/datake/FAME.