Deep Reinforcement Learning with Plasticity Injection

TL;DR

This paper introduces plasticity injection, a simple method to enhance neural network plasticity in deep reinforcement learning, serving as both a diagnostic tool and a means to improve training efficiency, especially in environments with plasticity loss.

Contribution

It proposes plasticity injection as a novel intervention to diagnose and mitigate plasticity loss in deep RL networks without increasing parameters or biasing predictions.

Findings

Plasticity injection improves performance over alternative methods.

It identifies environments where plasticity loss causes performance plateaus.

The method enhances training efficiency in Atari environments.

Abstract

A growing body of evidence suggests that neural networks employed in deep reinforcement learning (RL) gradually lose their plasticity, the ability to learn from new data; however, the analysis and mitigation of this phenomenon is hampered by the complex relationship between plasticity, exploration, and performance in RL. This paper introduces plasticity injection, a minimalistic intervention that increases the network plasticity without changing the number of trainable parameters or biasing the predictions. The applications of this intervention are two-fold: first, as a diagnostic tool $\unicode{x2014}$ if injection increases the performance, we may conclude that an agent's network was losing its plasticity. This tool allows us to identify a subset of Atari environments where the lack of plasticity causes performance plateaus, motivating future studies on understanding and combating plasticity loss. Second, plasticity injection can be used to improve the computational efficiency of RL training if the agent has to re-learn from scratch due to exhausted plasticity or by growing the agent's network dynamically without compromising performance. The results on Atari show that plasticity injection attains stronger performance compared to alternative methods while being computationally efficient.