DIRIGENt: End-To-End Robotic Imitation of Human Demonstrations Based on a Diffusion Model

TL;DR

DIRIGENt is an end-to-end diffusion-based model enabling robots to imitate human demonstrations directly from visual input, improving imitation accuracy and learning efficiency without prior human-robot pose mappings.

Contribution

The paper introduces a novel dataset, a diffusion-based imitation model, and an end-to-end architecture that enhances robot imitation capabilities from human demonstrations.

Findings

Outperforms existing methods in generating robot joint configurations from RGB images.

Creates a dataset with natural human-robot pose pairs for better imitation.

Diffusion input reduces redundancy and search space in joint configuration generation.

Abstract

There has been substantial progress in humanoid robots, with new skills continuously being taught, ranging from navigation to manipulation. While these abilities may seem impressive, the teaching methods often remain inefficient. To enhance the process of teaching robots, we propose leveraging a mechanism effectively used by humans: teaching by demonstrating. In this paper, we introduce DIRIGENt (DIrect Robotic Imitation GENeration model), a novel end-to-end diffusion approach that directly generates joint values from observing human demonstrations, enabling a robot to imitate these actions without any existing mapping between it and humans. We create a dataset in which humans imitate a robot and then use this collected data to train a diffusion model that enables a robot to imitate humans. The following three aspects are the core of our contribution. First is our novel dataset with natural pairs between human and robot poses, allowing our approach to imitate humans accurately despite the gap between their anatomies. Second, the diffusion input to our model alleviates the challenge of redundant joint configurations, limiting the search space. And finally, our end-to-end architecture from perception to action leads to an improved learning capability. Through our experimental analysis, we show that combining these three aspects allows DIRIGENt to outperform existing state-of-the-art approaches in the field of generating joint values from RGB images.