A two-way translation system of Chinese sign language based on computer vision

TL;DR

This paper presents a real-time two-way translation system between Chinese sign language and text, utilizing computer vision and neural networks to achieve high accuracy and speed for deaf communication.

Contribution

It introduces a novel integration of a TSM-enhanced neural network and an improved BERT model for effective sign language recognition and translation.

Findings

Sign language recognition accuracy of 99.3%

Recognition speed of about 0.05 seconds

Generation time of about 1.3 seconds

Abstract

As the main means of communication for deaf people, sign language has a special grammatical order, so it is meaningful and valuable to develop a real-time translation system for sign language. In the research process, we added a TSM module to the lightweight neural network model for the large Chinese continuous sign language dataset . It effectively improves the network performance with high accuracy and fast recognition speed. At the same time, we improve the Bert-Base-Chinese model to divide Chinese sentences into words and mapping the natural word order to the statute sign language order, and finally use the corresponding word videos in the isolated sign language dataset to generate the sentence video, so as to achieve the function of text-to-sign language translation. In the last of our research we built a system with sign language recognition and translation functions, and conducted performance tests on the complete dataset. The sign language video recognition accuracy reached about 99.3% with a time of about 0.05 seconds, and the sign language generation video time was about 1.3 seconds. The sign language system has good performance performance and is feasible.