Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

TL;DR

Introducing static network sparsity via one-shot random pruning significantly enhances the scalability, efficiency, and robustness of deep reinforcement learning models across various scenarios, surpassing dense architectures.

Contribution

Demonstrates that simple one-shot random pruning of networks enables better scaling and training stability in deep reinforcement learning, challenging the need for complex modifications.

Findings

Sparse networks achieve higher parameter efficiency.

Sparse networks show increased resistance to optimization challenges.

Benefits are consistent across visual and streaming RL scenarios.

Abstract

Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motivating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.