SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning

TL;DR

SemGes introduces a novel method for generating semantically coherent co-speech gestures by integrating semantic information at multiple levels, improving realism and relevance compared to existing approaches.

Contribution

The paper presents a new approach that incorporates semantic grounding at both fine-grained and global levels for co-speech gesture generation, leveraging a vector-quantized variational autoencoder.

Findings

Outperforms state-of-the-art methods in objective metrics

Enhances realism and semantic coherence of generated gestures

Validated through extensive experiments and user studies

Abstract

Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics. The qualitative results of our model, code, dataset and pre-trained models can be viewed at https://semgesture.github.io/.