Neuroplastic Expansion in Deep Reinforcement Learning
Jiashun Liu, Johan Obando-Ceron, Aaron Courville, Ling Pan
TL;DR
This paper introduces Neuroplastic Expansion, a novel method that dynamically grows neural networks during training to maintain plasticity, improve adaptability, and outperform existing methods in reinforcement learning tasks.
Contribution
The paper proposes a dynamic network growth approach inspired by cortical expansion, addressing plasticity loss in reinforcement learning agents.
Findings
NE outperforms state-of-the-art methods in MuJoCo and DeepMind Control Suite environments.
NE effectively mitigates plasticity loss and enhances adaptability in complex tasks.
The approach enables continual learning and adaptation in dynamic environments.
Abstract
The loss of plasticity in learning agents, analogous to the solidification of neural pathways in biological brains, significantly impedes learning and adaptation in reinforcement learning due to its non-stationary nature. To address this fundamental challenge, we propose a novel approach, {\it Neuroplastic Expansion} (NE), inspired by cortical expansion in cognitive science. NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension. Our method is designed with three key components: (\textit{1}) elastic topology generation based on potential gradients, (\textit{2}) dormant neuron pruning to optimize network expressivity, and (\textit{3}) neuron consolidation via experience review to strike a balance in the plasticity-stability dilemma. Extensive experiments demonstrate that NE…
Peer Reviews
Decision·ICLR 2025 Poster
1.The paper is well-written. 2.The concept of Neuroplastic Expansion (NE) is well-motivated.
See questions.
The authors present a novel approach that improves plasticity for deep reinforcement learning methods. The approach seems effective and achieves better performance than many existing methods, such as layer normalization, ReDo, and plasticity injection in many environments. The authors provided an extensive experimental study of their method in different environments (MuJoCO Gym and DMC) and with different learning algorithms (DrQ and TD3).
- The paper significantly lacks mathematical rigor. Here are some examples that are representative of these inaccuracies, although they don’t constitute an exhaustive list: - It should be $\breve{\theta} \subset \theta$ not $\breve{\theta} \in \theta$. Or more precisely, $\breve{\theta}_l \subset \theta_l, \forall l \in \\{1,...,N\\}$, where $N$ is the number of layers in the network. - In line 213, $\mathbb{I_{grow}}$ is not defined well. It should be a list, but you assign it with two ra
1. The core idea of NE seems very promising in terms of lifelong learning: add new capacity to learn new information, remove useless/dead neurons, and prevent catastrophic forgetting. Connection to biology is also a big plus. 2. The necessity for each component was well explained (Figure 2,3,4). I found it especially interesting to see a proof of catastrophic forgetting in a Mujoco task.
1. The writing is sometimes not detailed enough and causes confusion (see questions and weaknesses below). 2. Some crucial design choices are not well justified and/or validated. 1. Neuron consolidation is proposed to prevent catastrophic forgetting, which often occurs late stage (as shown in Figure 4). However, the dynamic threshold they use to control the strength of consolidation plateaus to its lowest value (strongest consolidation) even before halfway through the training process (Figur
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeuroscience and Neural Engineering
MethodsPruning