Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

TL;DR

This paper introduces RialTo, a system that creates digital twins from minimal real-world data to enhance imitation learning with reinforcement learning, achieving robust robotic manipulation policies efficiently.

Contribution

RialTo's novel approach constructs digital twins from limited data and employs inverse distillation for efficient policy fine-tuning, reducing human supervision and unsafe data collection.

Findings

Over 67% increase in policy robustness across tasks

Effective real-to-sim-to-real transfer for manipulation policies

Minimal human intervention required for environment modeling

Abstract

Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/