Bukva: Russian Sign Language Alphabet

TL;DR

Bukva introduces the first comprehensive open-source video dataset for Russian Sign Language dactyl recognition, enabling improved recognition of static and dynamic signs with high accuracy using a TSM-based model.

Contribution

This work provides the first extensive, publicly available dataset for RSL dactyl recognition, including dynamic signs and diverse subjects, along with a real-time recognition model.

Findings

Achieved 83.6% top-1 accuracy on RSL dactyl recognition.

Created a dataset with 3,757 videos and over 101 samples per sign.

Demonstrated effective handling of static and dynamic signs with TSM.

Abstract

This paper investigates the recognition of the Russian fingerspelling alphabet, also known as the Russian Sign Language (RSL) dactyl. Dactyl is a component of sign languages where distinct hand movements represent individual letters of a written language. This method is used to spell words without specific signs, such as proper nouns or technical terms. The alphabet learning simulator is an essential isolated dactyl recognition application. There is a notable issue of data shortage in isolated dactyl recognition: existing Russian dactyl datasets lack subject heterogeneity, contain insufficient samples, or cover only static signs. We provide Bukva, the first full-fledged open-source video dataset for RSL dactyl recognition. It contains 3,757 videos with more than 101 samples for each RSL alphabet sign, including dynamic ones. We utilized crowdsourcing platforms to increase the subject's heterogeneity, resulting in the participation of 155 deaf and hard-of-hearing experts in the dataset creation. We use a TSM (Temporal Shift Module) block to handle static and dynamic signs effectively, achieving 83.6% top-1 accuracy with a real-time inference with CPU only. The dataset, demo code, and pre-trained models are publicly available.