FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations

TL;DR

FR-GESTURE is a novel RGBD dataset designed for gesture-based control of unmanned ground vehicles by first responders, facilitating research in AI-driven human-robot interaction during disaster response.

Contribution

The paper introduces the first dedicated gesture dataset for first responder UGV control, including data collection, command set, evaluation protocols, and baseline experiments.

Findings

Dataset contains 3312 RGBD pairs from 2 viewpoints and 7 distances.

Defined evaluation protocols for gesture recognition.

Baseline experiments provided for future improvement.

Abstract

The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: https://doi.org/10.5281/zenodo.18131333.