Multilingual Communication System with Deaf Individuals Utilizing Natural and Visual Languages

TL;DR

This paper introduces MUGCAT, a multilingual system that converts sign language gestures into expressive images and meaningful sentences, enhancing real-time communication for deaf individuals across different languages.

Contribution

The paper presents a novel system that reconstructs meaningful sentences from incomplete sign language gestures and generates universal expressive images, improving cross-lingual deaf communication.

Findings

System operates in real-time.

Generates expressive images from gestures.

Produces meaningful sentences from sign language fragments.

Abstract

According to the World Federation of the Deaf, more than two hundred sign languages exist. Therefore, it is challenging to understand deaf individuals, even proficient sign language users, resulting in a barrier between the deaf community and the rest of society. To bridge this language barrier, we propose a novel multilingual communication system, namely MUGCAT, to improve the communication efficiency of sign language users. By converting recognized specific hand gestures into expressive pictures, which is universal usage and language independence, our MUGCAT system significantly helps deaf people convey their thoughts. To overcome the limitation of sign language usage, which is mostly impossible to translate into complete sentences for ordinary people, we propose to reconstruct meaningful sentences from the incomplete translation of sign language. We also measure the semantic similarity of generated sentences with fragmented recognized hand gestures to keep the original meaning. Experimental results show that the proposed system can work in a real-time manner and synthesize exquisite stunning illustrations and meaningful sentences from a few hand gestures of sign language. This proves that our MUGCAT has promising potential in assisting deaf communication.