Learning Novel Skills from Language-Generated Demonstrations

TL;DR

This paper introduces DemoGen, a framework that enables robots to learn new skills from language instructions by generating demonstration videos, significantly improving learning efficiency and safety.

Contribution

DemoGen uniquely combines vision-language and video diffusion models to generate demonstration videos from language, reducing reliance on manual demonstrations.

Findings

Generated demonstrations are high-fidelity and reliable.

Skill learning algorithms achieve three times higher success rates.

Framework enables intuitive and safe acquisition of new skills.

Abstract

Robots are increasingly deployed across diverse domains to tackle tasks requiring novel skills. However, current robot learning algorithms for acquiring novel skills often rely on demonstration datasets or environment interactions, resulting in high labor costs and potential safety risks. To address these challenges, this study proposes DemoGen, a skill-learning framework that enables robots to acquire novel skills from natural language instructions. DemoGen leverages the vision-language model and the video diffusion model to generate demonstration videos of novel skills, which enabling robots to learn new skills effectively. Experimental evaluations in the MetaWorld simulation environments demonstrate the pipeline's capability to generate high-fidelity and reliable demonstrations. Using the generated demonstrations, various skill learning algorithms achieve an accomplishment rate three times the original on novel tasks. These results highlight a novel approach to robot learning, offering a foundation for the intuitive and intelligent acquisition of novel robotic skills. (Project website: https://aoqunjin.github.io/LNSLGD/)