Plasticity Loss in Deep Reinforcement Learning: A Survey

TL;DR

This survey reviews the concept of plasticity loss in deep reinforcement learning, its causes, effects, and mitigation strategies, highlighting gaps in current evaluation methods and suggesting future research directions.

Contribution

It provides a unified definition of plasticity loss, organizes over 50 mitigation strategies into a taxonomy, and analyzes current evaluation practices in the field.

Findings

Regularization techniques often outperform domain-specific methods.

Gaps exist in current evaluation practices for plasticity loss.

Understanding mechanisms of plasticity loss is crucial for future progress.

Abstract

Plasticity refers to a network's ability to adapt to changing data distributions, which is crucial for the successful training of deep reinforcement learning agents. Loss of plasticity causes performance plateaus and contributes to scaling failures, overestimation bias, and insufficient exploration. To deepen the understanding of plasticity loss, we propose a unified definition, examine its drivers and pathologies, and organize over 50 mitigation strategies into the first comprehensive taxonomy of the field. Our analysis shows gaps in current evaluation practices and reveals that general regularization techniques often outperform domain-specific interventions. Future research should prioritize understanding the mechanisms underlying plasticity loss.