Evaluation of Generative Models for Emotional 3D Animation Generation in VR

TL;DR

This study evaluates speech-driven 3D emotional animation models in VR using user-centric metrics, revealing strengths in emotion recognition but limitations in realism, naturalness, and enjoyment compared to real human expressions.

Contribution

It provides a comprehensive user study assessing emotional 3D animation models in VR, highlighting the importance of emotion modeling and user-centric evaluation metrics.

Findings

Emotion-specific models improve recognition accuracy.

Happy animations rated more realistic and natural than neutral.

Generative models lag behind reconstruction-based methods in facial expression quality.

Abstract

Social interactions incorporate nonverbal signals to convey emotions alongside speech, including facial expressions and body gestures. Generative models have demonstrated promising results in creating full-body nonverbal animations synchronized with speech; however, evaluations using statistical metrics in 2D settings fail to fully capture user-perceived emotions, limiting our understanding of model effectiveness. To address this, we evaluate emotional 3D animation generative models within a Virtual Reality (VR) environment, emphasizing user-centric metrics emotional arousal realism, naturalness, enjoyment, diversity, and interaction quality in a real-time human-agent interaction scenario. Through a user study (N=48), we examine perceived emotional quality for three state of the art speech-driven 3D animation methods across two emotions happiness (high arousal) and neutral (mid arousal). Additionally, we compare these generative models against real human expressions obtained via a reconstruction-based method to assess both their strengths and limitations and how closely they replicate real human facial and body expressions. Our results demonstrate that methods explicitly modeling emotions lead to higher recognition accuracy compared to those focusing solely on speech-driven synchrony. Users rated the realism and naturalness of happy animations significantly higher than those of neutral animations, highlighting the limitations of current generative models in handling subtle emotional states. Generative models underperformed compared to reconstruction-based methods in facial expression quality, and all methods received relatively low ratings for animation enjoyment and interaction quality, emphasizing the importance of incorporating user-centric evaluations into generative model development. Finally, participants positively recognized animation diversity across all generative models.