Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

TL;DR

This paper introduces TeLAPA, a continual reinforcement learning framework that maintains diverse, skill-aligned policy neighborhoods in a shared latent space to enhance plasticity and transferability across tasks.

Contribution

TeLAPA shifts continual RL from single-model preservation to maintaining behaviorally diverse policy neighborhoods for better reuse and adaptation.

Findings

TeLAPA learns more tasks successfully in MiniGrid CL.

It recovers competence faster after interference.

It retains higher performance across task sequences.

Abstract

Continual reinforcement learning must balance retention with adaptation, yet many methods still rely on \emph{single-model preservation}, committing to one evolving policy as the main reusable solution across tasks. Even when a previously successful policy is retained, it may no longer provide a reliable starting point for rapid adaptation after interference, reflecting a form of \emph{loss of plasticity} that single-policy preservation cannot address. Inspired by quality-diversity methods, we introduce \textsc{TeLAPA} (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives and maintains a shared latent space so that archived policies remain comparable and reusable under non-stationary drift. This perspective shifts continual RL from retaining isolated solutions to maintaining \emph{skill-aligned neighborhoods} with competent and behaviorally related policies that support future relearning. In our MiniGrid CL setting, \textsc{TeLAPA} learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks. Our analyses show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative. Together, these results reframe continual RL around reusable and competent policy neighborhoods, providing a route beyond single-model preservation toward more plastic lifelong agents.