Lifelong Reinforcement Learning with Modulating Masks

TL;DR

This paper introduces modulating masks for deep lifelong reinforcement learning, enabling better knowledge retention, faster learning, and solving complex tasks with sparse rewards, outperforming existing baselines.

Contribution

It adapts modulating masks to deep RL agents like PPO and IMPALA, demonstrating improved performance and knowledge reuse in lifelong learning scenarios.

Findings

Superior performance over baselines in discrete and continuous tasks

Faster learning when combining previous masks

Ability to solve tasks with sparse rewards that were previously unsolvable

Abstract

Lifelong learning aims to create AI systems that continuously and incrementally learn during a lifetime, similar to biological learning. Attempts so far have met problems, including catastrophic forgetting, interference among tasks, and the inability to exploit previous knowledge. While considerable research has focused on learning multiple supervised classification tasks that involve changes in the input distribution, lifelong reinforcement learning (LRL) must deal with variations in the state and transition distributions, and in the reward functions. Modulating masks with a fixed backbone network, recently developed for classification, are particularly suitable to deal with such a large spectrum of task variations. In this paper, we adapted modulating masks to work with deep LRL, specifically PPO and IMPALA agents. The comparison with LRL baselines in both discrete and continuous RL tasks shows superior performance. We further investigated the use of a linear combination of previously learned masks to exploit previous knowledge when learning new tasks: not only is learning faster, the algorithm solves tasks that we could not otherwise solve from scratch due to extremely sparse rewards. The results suggest that RL with modulating masks is a promising approach to lifelong learning, to the composition of knowledge to learn increasingly complex tasks, and to knowledge reuse for efficient and faster learning.