Augmented Co-Speech Gesture Generation: Including Form and Meaning Features to Guide Learning-Based Gesture Synthesis

TL;DR

This paper introduces an augmented deep learning framework for co-speech gesture generation that incorporates form and meaning features, improving semantic coherence and person perception in artificial agents.

Contribution

It presents a novel approach that integrates semantic features into gesture synthesis, using small semantically annotated data and larger unannotated data for training.

Findings

Achieves semantic coherence comparable to human gestures

Demonstrates improved person perception in generated gestures

Provides publicly available data pipeline and framework

Abstract

Due to their significance in human communication, the automatic generation of co-speech gestures in artificial embodied agents has received a lot of attention. Although modern deep learning approaches can generate realistic-looking conversational gestures from spoken language, they often lack the ability to convey meaningful information and generate contextually appropriate gestures. This paper presents an augmented approach to the generation of co-speech gestures that additionally takes into account given form and meaning features for the gestures. Our framework effectively acquires this information from a small corpus with rich semantic annotations and a larger corpus without such information. We provide an analysis of the effects of distinctive feature targets and we report on a human rater evaluation study demonstrating that our framework achieves semantic coherence and person perception on the same level as human ground truth behavior. We make our data pipeline and the generation framework publicly available.