Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning

TL;DR

This paper introduces staggered environment resets in massively parallel on-policy RL to reduce nonstationarity, leading to improved sample efficiency, faster convergence, and better performance in high-dimensional robotics tasks.

Contribution

The paper proposes a simple staggered reset technique that mitigates nonstationarity in parallel RL environments, enhancing training stability and scalability.

Findings

Staggered resets improve sample efficiency in robotics environments.

They lead to faster convergence and stronger final performance.

The method scales better with more parallel environments.

Abstract

Massively parallel GPU simulation environments have accelerated reinforcement learning (RL) research by enabling fast data collection for on-policy RL algorithms like Proximal Policy Optimization (PPO). To maximize throughput, it is common to use short rollouts per policy update, increasing the update-to-data (UTD) ra- tio. However, we find that, in this setting, standard synchronous resets introduce harmful nonstationarity, skewing the learning signal and destabilizing training. We introduce staggered resets, a simple yet effective technique where environments are initialized and reset at varied points within the task horizon. This yields training batches with greater temporal diversity, reducing the nonstationarity induced by synchronized rollouts. We characterize dimensions along which RL environments can benefit significantly from staggered resets through illustrative toy environ- ments. We then apply this technique to challenging high-dimensional robotics environments, achieving significantly higher sample efficiency, faster wall-clock convergence, and stronger final performance. Finally, this technique scales better with more parallel environments compared to naive synchronized rollouts.