A Simulation Pipeline to Facilitate Real-World Robotic Reinforcement Learning Applications

TL;DR

This paper introduces a structured simulation pipeline that progressively enhances RL training stages to minimize the reality gap, enabling safer and more effective deployment of robotic policies in real-world applications.

Contribution

The paper proposes a novel RL training pipeline with multiple simulation stages, including system identification and high-fidelity simulation, to improve sim-to-real transfer for robotics.

Findings

Successful reduction of the reality gap in a Boston Dynamics Spot robot case study

Enhanced RL policy performance through staged simulation training

Demonstrated feasibility of deploying RL policies in real-world robotic systems

Abstract

Reinforcement learning (RL) has gained traction for its success in solving complex tasks for robotic applications. However, its deployment on physical robots remains challenging due to safety risks and the comparatively high costs of training. To avoid these problems, RL agents are often trained on simulators, which introduces a new problem related to the gap between simulation and reality. This paper presents an RL pipeline designed to help reduce the reality gap and facilitate developing and deploying RL policies for real-world robotic systems. The pipeline organizes the RL training process into an initial step for system identification and three training stages: core simulation training, high-fidelity simulation, and real-world deployment, each adding levels of realism to reduce the sim-to-real gap. Each training stage takes an input policy, improves it, and either passes the improved policy to the next stage or loops it back for further improvement. This iterative process continues until the policy achieves the desired performance. The pipeline's effectiveness is shown through a case study with the Boston Dynamics Spot mobile robot used in a surveillance application. The case study presents the steps taken at each pipeline stage to obtain an RL agent to control the robot's position and orientation.