On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective

TL;DR

This paper critically analyzes a gesture recognition evaluation protocol for UAV rescue operations, revealing that improper data splitting leads to overestimated accuracy and emphasizing the need for subject-independent testing to ensure real-world applicability.

Contribution

It identifies flaws in existing evaluation methods and advocates for subject-independent data partitioning to improve gesture recognition reliability in UAV applications.

Findings

Frame-level random train-test split causes data leakage.

Reported high accuracy does not reflect true generalization.

Subject-independent evaluation is essential for real-world UAV gesture recognition.

Abstract

This paper presents a methodological analysis of the gesture-recognition approach proposed by Liu and Szir\'anyi, with a particular focus on the validity of their evaluation protocol. We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split that inevitably mixes samples from the same subjects across both sets, causing severe data leakage. By examining the published confusion matrix, learning curves, and dataset construction, we demonstrate that the evaluation does not measure generalization to unseen individuals. Our findings underscore the importance of subject-independent data partitioning in vision-based gesture-recognition research, especially for applications - such as UAV-human interaction - that require reliable recognition of gestures performed by previously unseen people.