Markerless Visual Robot Programming by Demonstration
Raphael Memmesheimer, Ivanna Mykhalchyshyna, Viktor Seib, Nick, Theisen, Dietrich Paulus

TL;DR
This paper introduces a method for teaching robots to imitate human actions in a kitchen using markerless visual observation and scene understanding, enabling robots to learn tasks by demonstration without markers.
Contribution
The approach combines convolutional pose estimation, scene ontology, and spatial constraints to translate human demonstrations into executable robot commands.
Findings
Successfully learned a kitchen task through demonstration
Achieved markerless human pose and object detection in real-time
Enabled robot to imitate human behavior based on visual cues
Abstract
In this paper we present an approach for learning to imitate human behavior on a semantic level by markerless visual observation. We analyze a set of spatial constraints on human pose data extracted using convolutional pose machines and object informations extracted from 2D image sequences. A scene analysis, based on an ontology of objects and affordances, is combined with continuous human pose estimation and spatial object relations. Using a set of constraints we associate the observed human actions with a set of executable robot commands. We demonstrate our approach in a kitchen task, where the robot learns to prepare a meal.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobot Manipulation and Learning · Robotics and Sensor-Based Localization · Human Pose and Action Recognition
