Interpretation of Swedish Sign Language using Convolutional Neural Networks and Transfer Learning

TL;DR

This paper demonstrates that convolutional neural networks combined with transfer learning can effectively interpret Swedish Sign Language signs, achieving 85% accuracy with a small dataset, and presents a user-friendly web application.

Contribution

It introduces a CNN-based model using transfer learning for Swedish Sign Language interpretation and details its implementation as a web application.

Findings

Achieved 85% accuracy on sign language interpretation

Transfer learning enables high accuracy with limited data

Model implementation is accessible via a web application

Abstract

The automatic interpretation of sign languages is a challenging task, as it requires the usage of high-level vision and high-level motion processing systems for providing accurate image perception. In this paper, we use Convolutional Neural Networks (CNNs) and transfer learning in order to make computers able to interpret signs of the Swedish Sign Language (SSL) hand alphabet. Our model consist of the implementation of a pre-trained InceptionV3 network, and the usage of the mini-batch gradient descent optimization algorithm. We rely on transfer learning during the pre-training of the model and its data. The final accuracy of the model, based on 8 study subjects and 9,400 images, is 85%. Our results indicate that the usage of CNNs is a promising approach to interpret sign languages, and transfer learning can be used to achieve high testing accuracy despite using a small training dataset. Furthermore, we describe the implementation details of our model to interpret signs as a user-friendly web application.