AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learning

TL;DR

AltNet introduces a twin-network reinforcement learning approach that restores neural plasticity without performance drops, enhancing learning stability and efficiency in high-dimensional control tasks.

Contribution

We propose AltNet, a novel reset-based method using twin networks to maintain performance during plasticity restoration in reinforcement learning.

Findings

AltNet outperforms baseline methods in high-dimensional control tasks.

It achieves higher sample efficiency and stability.

AltNet avoids performance drops during resets.

Abstract

Artificial neural networks have shown remarkable success in supervised learning when trained on a single task using a fixed dataset. However, when neural networks are trained on a reinforcement learning task, their ability to continue learning from new experiences declines over time. This decline in learning ability is known as plasticity loss. To restore plasticity, prior work has explored periodically resetting the parameters of the learning network, a strategy that often improves performance. However, such resets come at the cost of a temporary drop in performance, which can be dangerous in real-world settings. To overcome this instability, we introduce AltNet, a reset-based approach that restores plasticity without performance degradation by leveraging a pair of twin networks. The use of twin networks anchors performance during resets through a mechanism that allows networks to periodically alternate roles: one network learns as it acts in the environment, while the other learns off-policy from the active network's interactions through a replay buffer. At fixed intervals, the active network is reset and the passive network, having learned from prior experience, becomes the new active network. AltNet restores plasticity, improving sample efficiency and achieving higher performance, while avoiding performance drops that pose risks in safety-critical settings. We demonstrate these advantages in several high-dimensional control tasks from the DeepMind Control Suite, where AltNet outperforms various relevant baseline methods, as well as state-of-the-art reset-based techniques.