CaptAinGlove: Capacitive and Inertial Fusion-Based Glove for Real-Time on Edge Hand Gesture Recognition for Drone Control

TL;DR

CaptAinGlove is a low-power, real-time glove-based system that uses multimodal fusion and lightweight neural networks to recognize hand gestures for drone control with high accuracy.

Contribution

This work introduces a textile-based, privacy-conscious glove with hierarchical multimodal fusion and lightweight CNNs for efficient on-the-edge gesture recognition.

Findings

Achieved 80% F1-score offline for nine gestures

Obtained 67% F1-score in real-time for one user

Low power consumption of 1.15 Watts

Abstract

We present CaptAinGlove, a textile-based, low-power (1.15Watts), privacy-conscious, real-time on-the-edge (RTE) glove-based solution with a tiny memory footprint (2MB), designed to recognize hand gestures used for drone control. We employ lightweight convolutional neural networks as the backbone models and a hierarchical multimodal fusion to reduce power consumption and improve accuracy. The system yields an F1-score of 80% for the offline evaluation of nine classes; eight hand gesture commands and null activity. For the RTE, we obtained an F1-score of 67% (one user).