Multimodal Robot Programming by Demonstration: A Preliminary Exploration

TL;DR

This paper explores the integration of multiple human modalities like gaze and speech into kinesthetic teaching for robots, aiming to improve robot learning and user programming in collaborative settings.

Contribution

It introduces the concept of multimodal demonstrations in robot programming by demonstration, highlighting potential benefits over traditional kinesthetic methods.

Findings

Multimodal demonstrations can clarify human intent.

Incorporating multiple modalities may reduce ambiguity in robot learning.

Preliminary study suggests potential for improved programming experiences.

Abstract

Recent years have seen a growth in the number of industrial robots working closely with end-users such as factory workers. This growing use of collaborative robots has been enabled in part due to the availability of end-user robot programming methods that allow users who are not robot programmers to teach robots task actions. Programming by Demonstration (PbD) is one such end-user programming method that enables users to bypass the complexities of specifying robot motions using programming languages by instead demonstrating the desired robot behavior. Demonstrations are often provided by physically guiding the robot through the motions required for a task action in a process known as kinesthetic teaching. Kinesthetic teaching enables users to directly demonstrate task behaviors in the robot's configuration space, making it a popular end-user robot programming method for collaborative robots known for its low cognitive burden. However, because kinesthetic teaching restricts the programmer's teaching to motion demonstrations, it fails to leverage information from other modalities that humans naturally use when providing physical task demonstrations to one other, such as gaze and speech. Incorporating multimodal information into the traditional kinesthetic programming workflow has the potential to enhance robot learning by highlighting critical aspects of a program, reducing ambiguity, and improving situational awareness for the robot learner and can provide insight into the human programmer's intent and difficulties. In this extended abstract, we describe a preliminary study on multimodal kinesthetic demonstrations and future directions for using multimodal demonstrations to enhance robot learning and user programming experiences.