Forager: a lightweight testbed for continual learning with partial observability in RL

TL;DR

Forager is a lightweight, partially observable reinforcement learning environment designed to facilitate continual learning research, highlighting challenges and potential mitigation strategies for agents with memory constraints.

Contribution

Introduces Forager, a novel, efficient CRL environment with constant memory, enabling in-depth study of partial observability and agent plasticity in continual learning.

Findings

Agents exhibit loss of plasticity in Forager.

Memory-based state construction improves agent performance.

Unending task stream exposes limitations of current CRL agents.

Abstract

In continual reinforcement learning (CRL), good performance requires never-ending learning, acting, and exploration in a big, partially observable world. Most CRL experiments have focused on loss of plasticity -- the inability to keep learning -- in one-off experiments where some unobservable non-stationarity is added to classic fully observable MDPs. Further, these experiments rarely consider the role of partial observability and the importance of CRL agents that use memory or recurrence. One potential reason for this focus on mitigating loss of plasticity without considering partial observability is that many partially-observable CRL environments are prohibitively expensive. In this paper, we introduce Forager, a light-weight partially-observable CRL environment with a constant memory footprint. We provide a set of experiments and sample tasks demonstrating that Forager is challenging for current CRL agents and yet also allows for in-depth study of those agents. We demonstrate that agents exhibit loss of plasticity, proposed mitigations can help, but that most useful is to leverage state construction. We conclude with a variant of Forager that generates an unending stream of new tasks to learn that clearly highlights the limitations of current CRL agents.