Empowering Sign Language Communication: Integrating Sentiment and Semantics for Facial Expression Synthesis

TL;DR

This paper presents a novel approach to sign language synthesis that incorporates sentiment and semantics to generate more expressive facial expressions, enhancing communication for deaf and hard-of-hearing individuals.

Contribution

The authors introduce a new method for synthesizing facial expressions in sign language by integrating sentiment and semantic features, along with a novel evaluation metric, achieving state-of-the-art results.

Findings

Achieved superior performance on How2Sign and PHOENIX14T datasets.

Introduced the Frechet Expression Distance (FED) metric for facial expression quality assessment.

Designed a graph pyramid architecture that simplifies training and enhances emotion-driven expression synthesis.

Abstract

Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker's emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speakers expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Frechet Gesture Distance (FGD) and propose a new metric called Frechet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions.