RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control

TL;DR

RMBench is a new benchmark for evaluating deep reinforcement learning algorithms on robotic manipulation tasks with high-dimensional sensory inputs, revealing current limitations and strengths of various algorithms.

Contribution

This paper introduces RMBench, the first comprehensive benchmark for deep RL in robotic manipulation, including implementation, evaluation, and analysis of multiple algorithms.

Findings

Soft Actor-Critic outperforms others in average reward and stability.

None of the studied algorithms handle all tasks well.

Data augmentation can improve policy learning.

Abstract

Reinforcement learning is applied to solve actual complex tasks from high-dimensional, sensory inputs. The last decade has developed a long list of reinforcement learning algorithms. Recent progress benefits from deep learning for raw sensory signal representation. One question naturally arises: how well do they perform concerning different robotic manipulation tasks? Benchmarks use objective performance metrics to offer a scientific way to compare algorithms. In this paper, we present RMBench, the first benchmark for robotic manipulations, which have high-dimensional continuous action and state spaces. We implement and evaluate reinforcement learning algorithms that directly use observed pixels as inputs. We report their average performance and learning curves to show their performance and stability of training. Our study concludes that none of the studied algorithms can handle all tasks well, soft Actor-Critic outperforms most algorithms in average reward and stability, and an algorithm combined with data augmentation may facilitate learning policies. Our code is publicly available at https://github.com/xiangyanfei212/RMBench-2022, including all benchmark tasks and studied algorithms.