Gesture-based Human-robot Interaction for Field Programmable Autonomous Underwater Robots

TL;DR

This paper presents a real-time gesture recognition system using CNNs for human-robot interaction underwater, enabling in situ programming of autonomous robots via customized sign language in challenging environments.

Contribution

It introduces a novel gesture-based interaction scheme and a CNN-based recognition system tailored for underwater environments, facilitating real-time human-robot collaboration.

Findings

Successfully recognized 50 gestures from monocular camera images

Demonstrated effective in situ underwater robot programming

Validated system performance through field trials

Abstract

The uncertainty and variability of underwater environment propose the request to control underwater robots in real time and dynamically, especially in the scenarios where human and robots need to work collaboratively in the field. However, the underwater environment imposes harsh restrictions on the application of typical control and communication methods. Considering that gestures are a natural and efficient interactive way for human, we, utilizing convolution neural network, implement a real-time gesture-based recognition system, who can recognize 50 kinds of gestures from images captured by one normal monocular camera, and apply this recognition system in human and underwater robot interaction. We design A Flexible and Extendable Interaction Scheme (AFEIS) through which underwater robots can be programmed in situ underwater by human operators using customized gesture-based sign language. This paper elaborates the design of gesture recognition system and AFEIS, and presents our field trial results when applying this system and scheme on underwater robots.