# Talking With Your Hands: Scaling Hand Gestures and Recognition With CNNs

## TL;DR

This paper introduces a CNN-based framework for recognizing scaled hand gestures formed by gesture-phonemes, enabling exponential growth in gesture vocabulary, supported by a new benchmark dataset and achieving high accuracy.

## Contribution

It proposes a novel methodology to scale hand gestures using gesture-phonemes and introduces the SHGD dataset for training and evaluation.

## Key findings

- Achieved 98.47% accuracy on 15-class gestures
- Achieved 94.69% accuracy on 810-class gestures
- Demonstrated exponential scaling of gesture vocabulary

## Abstract

The use of hand gestures provides a natural alternative to cumbersome interface devices for Human-Computer Interaction (HCI) systems. As the technology advances and communication between humans and machines becomes more complex, HCI systems should also be scaled accordingly in order to accommodate the introduced complexities. In this paper, we propose a methodology to scale hand gestures by forming them with predefined gesture-phonemes, and a convolutional neural network (CNN) based framework to recognize hand gestures by learning only their constituents of gesture-phonemes. The total number of possible hand gestures can be increased exponentially by increasing the number of used gesture-phonemes. For this objective, we introduce a new benchmark dataset named Scaled Hand Gestures Dataset (SHGD) with only gesture-phonemes in its training set and 3-tuples gestures in the test set. In our experimental analysis, we achieve to recognize hand gestures containing one and three gesture-phonemes with an accuracy of 98.47% (in 15 classes) and 94.69% (in 810 classes), respectively. Our dataset, code and pretrained models are publicly available.