An Advanced Deep Learning Based Three-Stream Hybrid Model for Dynamic Hand Gesture Recognition

TL;DR

This paper introduces a novel three-stream hybrid deep learning model that combines RGB pixel features and skeleton-based features for improved dynamic hand gesture recognition, addressing challenges like dataset limitations and environmental variations.

Contribution

The proposed model uniquely integrates multi-feature fusion from RGB images and hand pose key points using pre-trained and custom deep learning modules for enhanced gesture recognition.

Findings

Achieved high accuracy in hand gesture classification.

Effectively handled variations in hand forms and environmental conditions.

Demonstrated robustness through extensive dataset augmentation.

Abstract

In the modern context, hand gesture recognition has emerged as a focal point. This is due to its wide range of applications, which include comprehending sign language, factories, hands-free devices, and guiding robots. Many researchers have attempted to develop more effective techniques for recognizing these hand gestures. However, there are challenges like dataset limitations, variations in hand forms, external environments, and inconsistent lighting conditions. To address these challenges, we proposed a novel three-stream hybrid model that combines RGB pixel and skeleton-based features to recognize hand gestures. In the procedure, we preprocessed the dataset, including augmentation, to make rotation, translation, and scaling independent systems. We employed a three-stream hybrid model to extract the multi-feature fusion using the power of the deep learning module. In the first stream, we extracted the initial feature using the pre-trained Imagenet module and then enhanced this feature by using a multi-layer of the GRU and LSTM modules. In the second stream, we extracted the initial feature with the pre-trained ReseNet module and enhanced it with the various combinations of the GRU and LSTM modules. In the third stream, we extracted the hand pose key points using the media pipe and then enhanced them using the stacked LSTM to produce the hierarchical feature. After that, we concatenated the three features to produce the final. Finally, we employed a classification module to produce the probabilistic map to generate predicted output. We mainly produced a powerful feature vector by taking advantage of the pixel-based deep learning feature and pos-estimation-based stacked deep learning feature, including a pre-trained model with a scratched deep learning model for unequalled gesture detection capabilities.