EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generation

TL;DR

EASL is a novel sign language generation framework that incorporates multi-emotion guidance through semantic disentanglement, resulting in more natural and expressive sign language videos.

Contribution

The paper introduces emotion-semantic disentanglement modules and a multi-emotion-guided architecture for enhanced expressive sign language generation.

Findings

EASL outperforms baselines in pose accuracy.

EASL effectively integrates multi-emotion information.

Generated videos show improved expressiveness.

Abstract

Large language models have revolutionized sign language generation by automatically transforming text into high-quality sign language videos, providing accessible communication for the Deaf community. However, existing LLM-based approaches prioritize semantic accuracy while overlooking emotional expressions, resulting in outputs that lack naturalness and expressiveness. We propose EASL (Emotion-Aware Sign Language), a multi-emotion-guided generation architecture for fine-grained emotional integration. We introduce emotion-semantic disentanglement modules with progressive training to separately extract semantic and affective features. During pose decoding, the emotional representations guide semantic interaction to generate sign poses with 7-class emotion confidence scores, enabling emotional expression recognition. Experimental results demonstrate that EASL achieves pose accuracy superior to all compared baselines by integrating multi-emotion information and effectively adapts to diffusion models to generate expressive sign language videos.