Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning

TL;DR

This paper investigates how sparsification techniques like GMP and SET can mitigate plasticity loss in multi-task reinforcement learning, leading to improved adaptability and performance across various architectures and benchmarks.

Contribution

It systematically demonstrates that sparsification methods enhance plasticity and multi-task performance, providing new insights into their role in MTRL systems.

Findings

GMP and SET reduce neuron dormancy and representational collapse.

Sparse agents often outperform dense baselines in MTRL tasks.

Sparsification improves adaptability and performance in diverse MTRL architectures.

Abstract

Plasticity loss, a diminishing capacity to adapt as training progresses, is a critical challenge in deep reinforcement learning. We examine this issue in multi-task reinforcement learning (MTRL), where higher representational flexibility is crucial for managing diverse and potentially conflicting task demands. We systematically explore how sparsification methods, particularly Gradual Magnitude Pruning (GMP) and Sparse Evolutionary Training (SET), enhance plasticity and consequently improve performance in MTRL agents. We evaluate these approaches across distinct MTRL architectures (shared backbone, Mixture of Experts, Mixture of Orthogonal Experts) on standardized MTRL benchmarks, comparing against dense baselines, and a comprehensive range of alternative plasticity-inducing or regularization methods. Our results demonstrate that both GMP and SET effectively mitigate key indicators of plasticity degradation, such as neuron dormancy and representational collapse. These plasticity improvements often correlate with enhanced multi-task performance, with sparse agents frequently outperforming dense counterparts and achieving competitive results against explicit plasticity interventions. Our findings offer insights into the interplay between plasticity, network sparsity, and MTRL designs, highlighting dynamic sparsification as a robust but context-sensitive tool for developing more adaptable MTRL systems.