SLAC: Simulation-Pretrained Latent Action Space for Whole-Body Real-World RL

TL;DR

SLAC introduces a simulation-pretrained latent action space that enables efficient, safe, and effective real-world reinforcement learning for complex high-DoF robots, achieving state-of-the-art results in bimanual manipulation tasks with minimal real-world interactions.

Contribution

SLAC presents a novel approach combining low-fidelity simulation pretraining with unsupervised skill discovery to facilitate real-world RL on complex robots, reducing data requirements and improving safety.

Findings

Achieves state-of-the-art performance on manipulation tasks

Learns contact-rich whole-body tasks in under an hour

Does not require demonstrations or hand-crafted priors

Abstract

Building capable household and industrial robots requires mastering the control of versatile, high-degree-of-freedom (DoF) systems such as mobile manipulators. While reinforcement learning (RL) holds promise for autonomously acquiring robot control policies, scaling it to high-DoF embodiments remains challenging. Direct RL in the real world demands both safe exploration and high sample efficiency, which are difficult to achieve in practice. Sim-to-real RL, on the other hand, is often brittle due to the reality gap. This paper introduces SLAC, a method that renders real-world RL feasible for complex embodiments by leveraging a low-fidelity simulator to pretrain a task-agnostic latent action space. SLAC trains this latent action space via a customized unsupervised skill discovery method designed to promote temporal abstraction, disentanglement, and safety, thereby facilitating efficient downstream learning. Once a latent action space is learned, SLAC uses it as the action interface for a novel off-policy RL algorithm to autonomously learn downstream tasks through real-world interactions. We evaluate SLAC against existing methods on a suite of bimanual mobile manipulation tasks, where it achieves state-of-the-art performance. Notably, SLAC learns contact-rich whole-body tasks in under an hour of real-world interactions, without relying on any demonstrations or hand-crafted behavior priors. More information and robot videos at robo-rl.github.io