UAV Control with Vision-based Hand Gesture Recognition over Edge-Computing

TL;DR

This paper introduces a robust vision-based hand gesture recognition system for UAV control that leverages hand landmarks and edge computing to achieve real-time performance in dynamic environments.

Contribution

It proposes a novel hand landmark-based gesture recognition method and an edge-computing framework to enhance UAV control accuracy and efficiency under challenging conditions.

Findings

Our method outperforms existing approaches in accuracy and noise resilience.

Edge computing enables real-time gesture recognition on UAVs.

The system is effective both in simulation and real-world tests.

Abstract

Gesture recognition presents a promising avenue for interfacing with unmanned aerial vehicles (UAVs) due to its intuitive nature and potential for precise interaction. This research conducts a comprehensive comparative analysis of vision-based hand gesture detection methodologies tailored for UAV Control. The existing gesture recognition approaches involving cropping, zooming, and color-based segmentation, do not work well for this kind of applications in dynamic conditions and suffer in performance with increasing distance and environmental noises. We propose to use a novel approach leveraging hand landmarks drawing and classification for gesture recognition based UAV control. With experimental results we show that our proposed method outperforms the other existing methods in terms of accuracy, noise resilience, and efficacy across varying distances, thus providing robust control decisions. However, implementing the deep learning based compute intensive gesture recognition algorithms on the UAV's onboard computer is significantly challenging in terms of performance. Hence, we propose to use a edge-computing based framework to offload the heavier computing tasks, thus achieving closed-loop real-time performance. With implementation over AirSim simulator as well as over a real-world UAV, we showcase the advantage of our end-to-end gesture recognition based UAV control system.