SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

TL;DR

SkillMimicGen automates the creation of large, diverse demonstration datasets from minimal human input, significantly enhancing robot skill learning and enabling effective sim-to-real transfer for complex manipulation tasks.

Contribution

The paper introduces SkillMimicGen, a novel automated demonstration generation system that improves data efficiency and policy performance in robot manipulation tasks.

Findings

Generated over 24K demonstrations from 60 human demos

Achieved 24% higher success rates in simulation tasks

Enabled zero-shot sim-to-real transfer for long-horizon tasks

Abstract

Imitation learning from human demonstrations is an effective paradigm for robot manipulation, but acquiring large datasets is costly and resource-intensive, especially for long-horizon tasks. To address this issue, we propose SkillMimicGen (SkillGen), an automated system for generating demonstration datasets from a few human demos. SkillGen segments human demos into manipulation skills, adapts these skills to new contexts, and stitches them together through free-space transit and transfer motion. We also propose a Hybrid Skill Policy (HSP) framework for learning skill initiation, control, and termination components from SkillGen datasets, enabling skills to be sequenced using motion planning at test-time. We demonstrate that SkillGen greatly improves data generation and policy learning performance over a state-of-the-art data generation framework, resulting in the capability to produce data for large scene variations, including clutter, and agents that are on average 24% more successful. We demonstrate the efficacy of SkillGen by generating over 24K demonstrations across 18 task variants in simulation from just 60 human demonstrations, and training proficient, often near-perfect, HSP agents. Finally, we apply SkillGen to 3 real-world manipulation tasks and also demonstrate zero-shot sim-to-real transfer on a long-horizon assembly task. Videos, and more at https://skillgen.github.io.