Setting up a Reinforcement Learning Task with a Real-World Robot

TL;DR

This paper explores the setup of reinforcement learning tasks with real-world robots, highlighting key elements that influence learning performance and proposing guidelines for reliable, repeatable experiments with a UR5 robotic arm.

Contribution

It identifies critical setup factors affecting RL performance on robots and provides practical guidelines to improve experiment reliability and reproducibility.

Findings

Learning performance is highly sensitive to task setup details.

Proper setup can enable reliable and repeatable RL experiments with robots.

Guidelines help avoid common pitfalls in real-world robot RL research.

Abstract

Reinforcement learning is a promising approach to developing hard-to-engineer adaptive solutions for complex and diverse robotic tasks. However, learning with real-world robots is often unreliable and difficult, which resulted in their low adoption in reinforcement learning research. This difficulty is worsened by the lack of guidelines for setting up learning tasks with robots. In this work, we develop a learning task with a UR5 robotic arm to bring to light some key elements of a task setup and study their contributions to the challenges with robots. We find that learning performance can be highly sensitive to the setup, and thus oversights and omissions in setup details can make effective learning, reproducibility, and fair comparison hard. Our study suggests some mitigating steps to help future experimenters avoid difficulties and pitfalls. We show that highly reliable and repeatable experiments can be performed in our setup, indicating the possibility of reinforcement learning research extensively based on real-world robots.