A Survey of Continual Reinforcement Learning

TL;DR

This survey comprehensively reviews continual reinforcement learning, discussing its core concepts, challenges, methodologies, and proposing a new taxonomy to categorize existing methods.

Contribution

It provides a detailed review and analysis of CRL research, introduces a novel taxonomy, and highlights future research directions.

Findings

Organized existing CRL works by metrics, tasks, benchmarks, and scenarios.

Proposed a new taxonomy categorizing CRL methods into four types.

Identified key challenges and practical insights for future CRL research.

Abstract

Reinforcement Learning (RL) is an important machine learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in this field due to the rapid development of deep neural networks. However, the success of RL currently relies on extensive training data and computational resources. In addition, RL's limited ability to generalize across tasks restricts its applicability in dynamic and real-world environments. With the arisen of Continual Learning (CL), Continual Reinforcement Learning (CRL) has emerged as a promising research direction to address these limitations by enabling agents to learn continuously, adapt to new tasks, and retain previously acquired knowledge. In this survey, we provide a comprehensive examination of CRL, focusing on its core concepts, challenges, and methodologies. Firstly, we conduct a detailed review of existing works, organizing and analyzing their metrics, tasks, benchmarks, and scenario settings. Secondly, we propose a new taxonomy of CRL methods, categorizing them into four types from the perspective of knowledge storage and/or transfer. Finally, our analysis highlights the unique challenges of CRL and provides practical insights into future directions.