Going into Orbit: Massively Parallelizing Episodic Reinforcement Learning

TL;DR

This paper demonstrates how NVIDIA's Orbit framework enables massively parallelized reinforcement learning training in simulation, significantly increasing sample throughput and efficiency compared to CPU-based methods.

Contribution

The paper provides a detailed implementation of a benchmark task using Orbit and benchmarks its performance against CPU-based approaches.

Findings

Orbit achieves higher sample throughput than CPU implementations.

Parallelization with Orbit reduces training time for reinforcement learning tasks.

Hyperparameter tuning further enhances sample generation efficiency.

Abstract

The possibilities of robot control have multiplied across various domains through the application of deep reinforcement learning. To overcome safety and sampling efficiency issues, deep reinforcement learning models can be trained in a simulation environment, allowing for faster iteration cycles. This can be enhanced further by parallelizing the training process using GPUs. NVIDIA's open-source robot learning framework Orbit leverages this potential by wrapping tensor-based reinforcement learning libraries for high parallelism and building upon Isaac Sim for its simulations. We contribute a detailed description of the implementation of a benchmark reinforcement learning task, namely box pushing, using Orbit. Additionally, we benchmark the performance of our implementation in comparison to a CPU-based implementation and report the performance metrics. Finally, we tune the hyper parameters of our implementation and show that we can generate significantly more samples in the same amount of time by using Orbit.