Implementation and Evaluation of multimodal input/output channels for task-based industrial robot programming

TL;DR

This paper presents the development and evaluation of a multimodal user interface for task-based industrial robot programming, enabling non-experts to program robots intuitively using gesture, speech, touch, or pen inputs.

Contribution

It introduces a flexible, intuitive GUI for robot programming that adapts input modalities based on task analysis and user study results.

Findings

Gesture, speech, touch, and pen inputs were evaluated for effectiveness.

Non-expert users successfully programmed robots using the multimodal interface.

The system can automatically infer suitable input modalities based on task context.

Abstract

Programming industrial robots is not very intuitive, and the programmer has to be a domain expert for e.g. welding and programming to know how the task is optimally executed. For SMEs such employees are not affordable, nor cost-effective. Therefore a new system is needed where domain experts from a specific area, like welding or assembly, can easily program a robot without knowing anything about programming languages or how to use TeachPads. Such a system needs to be flexible to adapt to new tasks and functions. These requirements can be met by using a task based programming approach where the robot program is built up using a hierarchical structure of process, tasks and skills. It also needs to be intuitive so that domain experts don't need much training time on handling the system. Intuitive interaction is achieved by using different input and output modalities like gesture input, speech input, or touch input which are suitable for the current task. This master thesis focuses on the implementation of a user interface (GUI) for task based industrial robot programming and evaluates different input modalities (gesture, speech, touch, pen input) for the interaction with the system. The evaluation is based on a user study conducted with 30 participants as a Wizard-Of-Oz experiment, where non expert users had to program assembly and welding tasks to an industrial robot, using the previously developed GUI and various input and output modalities. The findings of the task analysis and user study are then used for creating a semantic description which will be used in the cognitive robotics-worker cell for automatically inferring required system components, and to provide the best suited input modality.