Intentional Gesture: Deliver Your Intentions with Gestures for Speech

TL;DR

This paper presents Intentional-Gesture, a framework for generating semantically meaningful gestures aligned with speech by reasoning about high-level communicative intentions, advancing gesture synthesis in digital humans.

Contribution

It introduces a novel intention-aware gesture generation framework, including a new dataset with intention annotations and a motion tokenizer that incorporates high-level communicative functions.

Findings

Achieves state-of-the-art performance on BEAT-2 benchmark.

Demonstrates improved semantic relevance of generated gestures.

Provides a modular foundation for expressive gesture synthesis.

Abstract

When humans speak, gestures help convey communicative intentions, such as adding emphasis or describing concepts. However, current co-speech gesture generation methods rely solely on superficial linguistic cues (e.g. speech audio or text transcripts), neglecting to understand and leverage the communicative intention that underpins human gestures. This results in outputs that are rhythmically synchronized with speech but are semantically shallow. To address this gap, we introduce Intentional-Gesture, a novel framework that casts gesture generation as an intention-reasoning task grounded in high-level communicative functions. First, we curate the InG dataset by augmenting BEAT-2 with gesture-intention annotations (i.e., text sentences summarizing intentions), which are automatically annotated using large vision-language models. Next, we introduce the Intentional Gesture Motion Tokenizer to leverage these intention annotations. It injects high-level communicative functions (e.g., intentions) into tokenized motion representations to enable intention-aware gesture synthesis that are both temporally aligned and semantically meaningful, achieving new state-of-the-art performance on the BEAT-2 benchmark. Our framework offers a modular foundation for expressive gesture generation in digital humans and embodied AI. Project Page: https://andypinxinliu.github.io/Intentional-Gesture