Real Time Captioning of Sign Language Gestures in Video Meetings

TL;DR

This paper presents a browser extension that automatically translates sign language gestures into subtitles during video meetings, leveraging a large dataset of ASL videos to improve communication for the hearing impaired.

Contribution

It introduces a real-time sign language recognition system integrated into a browser extension for video conferencing, utilizing a new large-scale ASL dataset.

Findings

Effective real-time sign language translation demonstrated

Enhanced communication accessibility for hearing-impaired users

Large-scale dataset supports robust recognition

Abstract

It has always been a rather tough task to communicate with someone possessing a hearing impairment. One of the most tested ways to establish such a communication is through the use of sign based languages. However, not many people are aware of the smaller intricacies involved with sign language. Sign language recognition using computer vision aims at eliminating the communication barrier between deaf-mute and ordinary people so that they can properly communicate with others. Recently the pandemic has left the whole world shaken up and has transformed the way we communicate. Video meetings have become essential for everyone, even people with a hearing disability. In recent studies, it has been found that people with hearing disabilities prefer to sign over typing during these video calls. In this paper, we are proposing a browser extension that will automatically translate sign language to subtitles for everyone else in the video call. The Large-scale dataset which contains more than 2000 Word-Level ASL videos, which were performed by over 100 signers will be used.