PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning

TL;DR

This paper introduces the PLASTIC algorithm that enhances input and label plasticity in reinforcement learning, leading to improved sample efficiency by preventing overfitting and maintaining adaptability.

Contribution

The paper identifies two aspects of plasticity in RL, proposes techniques to improve them, and develops the PLASTIC algorithm that achieves competitive results on standard benchmarks.

Findings

Smoother loss minima improve input plasticity.

Refined gradient propagation enhances label plasticity.

PLASTIC achieves competitive performance on Atari-100k and DeepMind Control Suite.

Abstract

In Reinforcement Learning (RL), enhancing sample efficiency is crucial, particularly in scenarios when data acquisition is costly and risky. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. However, these multiple updates often lead the model to overfit to earlier interactions, which is referred to as the loss of plasticity. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects. Input plasticity, which denotes the model's adaptability to changing input data, and label plasticity, which denotes the model's adaptability to evolving input-output relationships. Synthetic experiments on the CIFAR-10 dataset reveal that finding smoother minima of loss landscape enhances input plasticity, whereas refined gradient propagation improves label plasticity. Leveraging these findings, we introduce the PLASTIC algorithm, which harmoniously combines techniques to address both concerns. With minimal architectural modifications, PLASTIC achieves competitive performance on benchmarks including Atari-100k and Deepmind Control Suite. This result emphasizes the importance of preserving the model's plasticity to elevate the sample efficiency in RL. The code is available at https://github.com/dojeon-ai/plastic.