GesSure- A Robust Face-Authentication enabled Dynamic Gesture Recognition GUI Application

TL;DR

This paper presents GesSure, a secure gesture recognition GUI that uses face verification for user authentication, achieving high accuracy and enabling intuitive, context-aware human-machine interactions without physical contact.

Contribution

The paper introduces a novel face-verification-enabled gesture recognition system combining MTCNN, FaceNet, and LSTM-CNN, enhancing security and user experience in HCI applications.

Findings

Achieved 95% gesture recognition accuracy.

Successfully executed context-dependent and context-free tasks.

Open-sourced application and dataset for community use.

Abstract

Using physical interactive devices like mouse and keyboards hinders naturalistic human-machine interaction and increases the probability of surface contact during a pandemic. Existing gesture-recognition systems do not possess user authentication, making them unreliable. Static gestures in current gesture-recognition technology introduce long adaptation periods and reduce user compatibility. Our technology places a strong emphasis on user recognition and safety. We use meaningful and relevant gestures for task operation, resulting in a better user experience. This paper aims to design a robust, face-verification-enabled gesture recognition system that utilizes a graphical user interface and primarily focuses on security through user recognition and authorization. The face model uses MTCNN and FaceNet to verify the user, and our LSTM-CNN architecture for gesture recognition, achieving an accuracy of 95% with five classes of gestures. The prototype developed through our research has successfully executed context-dependent tasks like save, print, control video-player operations and exit, and context-free operating system tasks like sleep, shut-down, and unlock intuitively. Our application and dataset are available as open source.