Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents

TL;DR

Text2Gestures is a transformer-based model that generates emotionally expressive, full-body gestures for virtual agents aligned with natural language, considering biomechanical features and agent attributes, achieving state-of-the-art results.

Contribution

The paper introduces a novel transformer-based approach that generates contextually and emotionally appropriate gestures for virtual agents, incorporating biomechanical and agent-specific features.

Findings

Achieves state-of-the-art gesture-text alignment performance.

Participants found generated gestures highly plausible.

Generated gestures strongly correlate with intended emotions.

Abstract

We present Text2Gestures, a transformer-based learning method to interactively generate emotive full-body gestures for virtual agents aligned with natural language text inputs. Our method generates emotionally expressive gestures by utilizing the relevant biomechanical features for body expressions, also known as affective features. We also consider the intended task corresponding to the text and the target virtual agents' intended gender and handedness in our generation pipeline. We train and evaluate our network on the MPI Emotional Body Expressions Database and observe that our network produces state-of-the-art performance in generating gestures for virtual agents aligned with the text for narration or conversation. Our network can generate these gestures at interactive rates on a commodity GPU. We conduct a web-based user study and observe that around 91% of participants indicated our generated gestures to be at least plausible on a five-point Likert Scale. The emotions perceived by the participants from the gestures are also strongly positively correlated with the corresponding intended emotions, with a minimum Pearson coefficient of 0.77 in the valence dimension.