Robotic Detection of a Human-Comprehensible Gestural Language for Underwater Multi-Human-Robot Collaboration

TL;DR

This paper introduces a visual gestural language and a deep learning system for underwater robots to communicate non-verbally with each other and with human divers, enabling more intuitive multi-human-robot collaboration.

Contribution

We developed a novel visual gestural language for underwater communication and a deep network, RRCommNet, that recognizes gestures with high accuracy in real-world conditions.

Findings

RRCommNet achieves 88-94% accuracy on simulated data

RRCommNet achieves 73-83% accuracy on real-world data

Humans understand the gestural language with 88% transcription accuracy

Abstract

In this paper, we present a motion-based robotic communication framework that enables non-verbal communication among autonomous underwater vehicles (AUVs) and human divers. We design a gestural language for AUV-to-AUV communication which can be easily understood by divers observing the conversation unlike typical radio frequency, light, or audio based AUV communication. To allow AUVs to visually understand a gesture from another AUV, we propose a deep network (RRCommNet) which exploits a self-attention mechanism to learn to recognize each message by extracting maximally discriminative spatio-temporal features. We train this network on diverse simulated and real-world data. Our experimental evaluations, both in simulation and in closed-water robot trials, demonstrate that the proposed RRCommNet architecture is able to decipher gesture-based messages with an average accuracy of 88-94% on simulated data, 73-83% on real data (depending on the version of the model used). Further, by performing a message transcription study with human participants, we also show that the proposed language can be understood by humans, with an overall transcription accuracy of 88%. Finally, we discuss the inference runtime of RRCommNet on embedded GPU hardware, for real-time use on board AUVs in the field.