Towards Decentralized Human-Swarm Interaction by Means of Sequential Hand Gesture Recognition

TL;DR

This paper introduces a decentralized human-swarm interaction method using hand gesture recognition with CNNs, enabling robots to modify their behavior based on gestures in simulation and real-world tests.

Contribution

It presents a novel approach combining CNN-based gesture recognition with decentralized robot control for human-swarm interaction.

Findings

Successful recognition of hand gestures in simulation and real-world environments.

Effective modification of swarm behavior based on gesture sequences.

Validation in Gazebo simulations and physical robot tests.

Abstract

In this work, we present preliminary work on a novel method for Human-Swarm Interaction (HSI) that can be used to change the macroscopic behavior of a swarm of robots with decentralized sensing and control. By integrating a small yet capable hand gesture recognition convolutional neural network (CNN) with the next-generation Robot Operating System \emph{ros2}, which enables decentralized implementation of robot software for multi-robot applications, we demonstrate the feasibility of programming a swarm of robots to recognize and respond to a sequence of hand gestures that capable of correspond to different types of swarm behaviors. We test our approach using a sequence of gestures that modifies the target inter-robot distance in a group of three Turtlebot3 Burger robots in order to prevent robot collisions with obstacles. The approach is validated in three different Gazebo simulation environments and in a physical testbed that reproduces one of the simulated environments.