System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games

TL;DR

This paper introduces a standardized framework and environment for integrating and evaluating multiple lifelong reinforcement learning components in real-time strategy games, advancing practical L2RL system development.

Contribution

It proposes the Lifelong Reinforcement Learning Components Framework (L2RLCF) and a standard API for system integration, enabling comprehensive evaluation of L2RL components.

Findings

Successful integration of multiple L2RL components into a unified system.

Development of a new evaluation environment for L2RL in Starcraft-2 minigames.

Facilitation of fair, comprehensive comparison of different L2RL approaches.

Abstract

As Artificial and Robotic Systems are increasingly deployed and relied upon for real-world applications, it is important that they exhibit the ability to continually learn and adapt in dynamically-changing environments, becoming Lifelong Learning Machines. Continual/lifelong learning (LL) involves minimizing catastrophic forgetting of old tasks while maximizing a model's capability to learn new tasks. This paper addresses the challenging lifelong reinforcement learning (L2RL) setting. Pushing the state-of-the-art forward in L2RL and making L2RL useful for practical applications requires more than developing individual L2RL algorithms; it requires making progress at the systems-level, especially research into the non-trivial problem of how to integrate multiple L2RL algorithms into a common framework. In this paper, we introduce the Lifelong Reinforcement Learning Components Framework (L2RLCF), which standardizes L2RL systems and assimilates different continual learning components (each addressing different aspects of the lifelong learning problem) into a unified system. As an instantiation of L2RLCF, we develop a standard API allowing easy integration of novel lifelong learning components. We describe a case study that demonstrates how multiple independently-developed LL components can be integrated into a single realized system. We also introduce an evaluation environment in order to measure the effect of combining various system components. Our evaluation environment employs different LL scenarios (sequences of tasks) consisting of Starcraft-2 minigames and allows for the fair, comprehensive, and quantitative comparison of different combinations of components within a challenging common evaluation environment.