Sentiment Analysis of German Sign Language Fairy Tales

TL;DR

This paper introduces a dataset and models for sentiment analysis of German sign language fairy tales, combining text and video features to understand emotional communication.

Contribution

It presents a novel multimodal approach using large language models and explainable video feature analysis for sentiment detection in sign language.

Findings

Achieved an inter-annotator agreement of 0.781 Krippendorff's alpha.

Reached an average balanced accuracy of 0.631 in sentiment prediction.

Identified face and body motion features as key contributors to sentiment discrimination.

Abstract

We present a dataset and a model for sentiment analysis of German sign language (DGS) fairy tales. First, we perform sentiment analysis for three levels of valence (negative, neutral, positive) on German fairy tales text segments using four large language models (LLMs) and majority voting, reaching an inter-annotator agreement of 0.781 Krippendorff's alpha. Second, we extract face and body motion features from each corresponding DGS video segment using MediaPipe. Finally, we train an explainable model (based on XGBoost) to predict negative, neutral or positive sentiment from video features. Results show an average balanced accuracy of 0.631. A thorough analysis of the most important features reveal that, in addition to eyebrows and mouth motion on the face, also the motion of hips, elbows, and shoulders considerably contribute in the discrimination of the conveyed sentiment, indicating an equal importance of face and body for sentiment communication in sign language.