Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretraining

TL;DR

This paper presents a cross-embodiment imitation learning system for quadrupedal robots, leveraging human data for pretraining to enhance manipulation skills across various household tasks, with significant success rate improvements.

Contribution

It introduces a novel modular architecture and a large manipulation dataset for quadruped robots, enabling effective human-to-robot transfer and pretraining for versatile manipulation skills.

Findings

41.9% average success rate improvement over baseline

82.7% success rate boost under out-of-distribution conditions with human pretraining

First manipulation dataset for LocoMan robot covering diverse household tasks

Abstract

Quadrupedal robots have demonstrated impressive locomotion capabilities in complex environments, but equipping them with autonomous versatile manipulation skills in a scalable way remains a significant challenge. In this work, we introduce a cross-embodiment imitation learning system for quadrupedal manipulation, leveraging data collected from both humans and LocoMan, a quadruped equipped with multiple manipulation modes. Specifically, we develop a teleoperation and data collection pipeline, which unifies and modularizes the observation and action spaces of the human and the robot. To effectively leverage the collected data, we propose an efficient modularized architecture that supports co-training and pretraining on structured modality-aligned data across different embodiments. Additionally, we construct the first manipulation dataset for the LocoMan robot, covering various household tasks in both unimanual and bimanual modes, supplemented by a corresponding human dataset. We validate our system on six real-world manipulation tasks, where it achieves an average success rate improvement of 41.9% overall and 79.7% under out-of-distribution (OOD) settings compared to the baseline. Pretraining with human data contributes a 38.6% success rate improvement overall and 82.7% under OOD settings, enabling consistently better performance with only half the amount of robot data. Our code, hardware, and data are open-sourced at: https://human2bots.github.io.