Emotion Recognition in Signers

TL;DR

This paper tackles emotion recognition in sign language by addressing data scarcity and grammatical-affective overlap, using cross-lingual datasets and demonstrating the benefits of text and hand motion cues for improved accuracy.

Contribution

It introduces the eJSL dataset for Japanese Sign Language, explores cross-lingual transfer, and shows how textual cues and hand motion improve emotion recognition in signers.

Findings

Textual emotion recognition reduces data scarcity issues.

Temporal segment selection significantly affects performance.

Incorporating hand motion improves emotion recognition accuracy.

Abstract

Recognition of signers' emotions suffers from one theoretical challenge and one practical challenge, namely, the overlap between grammatical and affective facial expressions and the scarcity of data for model training. This paper addresses these two challenges in a cross-lingual setting using our eJSL dataset, a new benchmark dataset for emotion recognition in Japanese Sign Language signers, and BOBSL, a large British Sign Language dataset with subtitles. In eJSL, two signers expressed 78 distinct utterances with each of seven different emotional states, resulting in 1,092 video clips. We empirically demonstrate that 1) textual emotion recognition in spoken language mitigates data scarcity in sign language, 2) temporal segment selection has a significant impact, and 3) incorporating hand motion enhances emotion recognition in signers. Finally we establish a stronger baseline than spoken language LLMs.