A novel deep transformer based CvT model for sign language recognition in visual communication

TL;DR

This paper introduces a new deep learning model for sign language recognition that outperforms existing methods using a combination of convolutional and transformer techniques.

Contribution

The novel contribution is a CvT model that integrates hierarchical convolutional tokenization with transformer attention for improved sign language recognition.

Findings

The proposed CvT model achieves 99% accuracy on a sign language digits dataset.

The model outperforms traditional CNN and BeIT transformer models in classification performance.

The model improves generalization and reduces misclassifications across training, validation, and test sets.

Abstract

Sign language serves as a crucial mode of communication for the deaf and hard-of-hearing communities, enabling effective interaction in daily life. With the growing advancements in Artificial Intelligence (AI) and computer vision, there has been a significant shift toward automating SLR, making communication more accessible and inclusive. Traditional AI-based approaches, such as rule-based and statistical models, struggle to handle complex hand gestures, varying lighting conditions, and occlusions. Deep learning-based methods, particularly Convolutional Neural Networks (CNNs), have improved recognition capabilities, but they often fail to capture intricate spatial and temporal dependencies that are essential for accurate classification. To address these limitations, vision transformers (ViTs) have emerged as a breakthrough technology, offering superior feature extraction through self-attention mechanisms. Unlike conventional CNNs, ViTs efficiently model long-range dependencies, enabling robust sign recognition. This study proposes a Convolutional Vision Transformer (CvT)-based model that integrates hierarchical convolutional tokenization with transformer-based attention mechanisms, optimizing both local and global feature extraction. The proposed CvT model was evaluated on a publicly available sign language digits dataset, consisting of 1,712 images across 10 different classes along with alphabet and symbol dataset with 87,000 images of 29 classes. Empirical results indicate that with both datasets, the proposed model analysis CvT outperforms baseline models, achieving the highest accuracy of 99%, surpassing traditional CNN and transformer-based BeIT models. The findings demonstrate that CvT effectively reduces misclassifications, improves predictive confidence, and enhances generalization across training, validation, and test sets.

Figures