Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Models

TL;DR

This paper introduces MECo, a novel framework that leverages large language models to generate co-speech gestures controlled by motion examples, preserving detailed motion characteristics and enabling diverse, granular, multimodal inputs.

Contribution

The paper presents a new approach using LLMs for motion-example-controlled gesture generation, moving beyond pseudo-labeling to explicit query-based guidance, achieving state-of-the-art results.

Findings

State-of-the-art performance on FGD, diversity, and similarity metrics

Supports granular control of individual body parts

Handles multiple input modalities including text and video

Abstract

The automatic generation of controllable co-speech gestures has recently gained growing attention. While existing systems typically achieve gesture control through predefined categorical labels or implicit pseudo-labels derived from motion examples, these approaches often compromise the rich details present in the original motion examples. We present MECo, a framework for motion-example-controlled co-speech gesture generation by leveraging large language models (LLMs). Our method capitalizes on LLMs' comprehension capabilities through fine-tuning to simultaneously interpret speech audio and motion examples, enabling the synthesis of gestures that preserve example-specific characteristics while maintaining speech congruence. Departing from conventional pseudo-labeling paradigms, we position motion examples as explicit query contexts within the prompt structure to guide gesture generation. Experimental results demonstrate state-of-the-art performance across three metrics: Fr\'echet Gesture Distance (FGD), motion diversity, and example-gesture similarity. Furthermore, our framework enables granular control of individual body parts and accommodates diverse input modalities including motion clips, static poses, human video sequences, and textual descriptions. Our code, pre-trained models, and videos are available at https://robinwitch.github.io/MECo-Page.