Rethinking Plasticity in Deep Reinforcement Learning

TL;DR

This paper introduces the Optimization-Centric Plasticity hypothesis, explaining plasticity loss in deep RL as a result of local optima entrenchment, and demonstrates how this understanding can guide strategies to restore network adaptability in non-stationary environments.

Contribution

It proposes a new optimization-based framework for understanding plasticity loss, linking neuron dormancy to zero-gradient states and explaining the effectiveness of parameter constraints.

Findings

Plasticity loss is task-specific and reversible.

Neuron dormancy correlates with zero-gradient states.

Parameter constraints help prevent local optima entrenchment.

Abstract

This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL), a critical challenge where neural networks lose their ability to adapt to non-stationary environments. While existing research often relies on descriptive metrics like dormant neurons or effective rank, these summaries fail to explain the underlying optimization dynamics. We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks, trapping parameters during task transitions and hindering subsequent learning. We theoretically establish the equivalence between neuron dormancy and zero-gradient states, demonstrating that the absence of gradient signals is the primary driver of dormancy. Our experiments reveal that plasticity loss is highly task-specific; notably, networks with high dormancy rates in one task can achieve performance parity with randomly initialized networks when switched to a significantly different task, suggesting that the network's capacity remains intact but is inhibited by the specific optimization landscape. Furthermore, our hypothesis elucidates why parameter constraints mitigate plasticity loss by preventing deep entrenchment in local optima. Validated across diverse non-stationary scenarios, our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex RL domains.