Evaluating Pointing Gestures for Target Selection in Human-Robot Collaboration

TL;DR

This paper presents a new method for localizing pointing gestures in human-robot collaboration using pose estimation and geometric modeling, enabling improved target selection and multimodal interaction.

Contribution

It introduces a novel approach for localizing pointing gestures with a geometric model and evaluates its integration into a multimodal robotic system.

Findings

Effective localization of pointing targets in a planar workspace

Successful integration of gesture recognition with object detection and speech modules

Analysis of tool limitations and performance in collaborative tasks

Abstract

Pointing gestures are a common interaction method used in Human-Robot Collaboration for various tasks, ranging from selecting targets to guiding industrial processes. This study introduces a method for localizing pointed targets within a planar workspace. The approach employs pose estimation, and a simple geometric model based on shoulder-wrist extension to extract gesturing data from an RGB-D stream. The study proposes a rigorous methodology and comprehensive analysis for evaluating pointing gestures and target selection in typical robotic tasks. In addition to evaluating tool accuracy, the tool is integrated into a proof-of-concept robotic system, which includes object detection, speech transcription, and speech synthesis to demonstrate the integration of multiple modalities in a collaborative application. Finally, a discussion over tool limitations and performance is provided to understand its role in multimodal robotic systems. All developments are available at: https://github.com/NMKsas/gesture_pointer.git.