TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation

TL;DR

TANGO is a novel framework that generates realistic co-speech gesture videos by combining hierarchical audio-motion embedding, diffusion-based transition frame generation, and a graph retrieval system to ensure synchronization and visual quality.

Contribution

It introduces a hierarchical joint embedding space (AuMoCLIP) for better cross-modal alignment and a diffusion-based model (ACInterp) for high-quality transition frames in gesture video reenactment.

Findings

Outperforms existing methods in realism and synchronization

Achieves high-fidelity, audio-synchronized gesture videos

Effectively reduces visual artifacts in generated videos

Abstract

We present TANGO, a framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video and target speech audio, TANGO produces high-fidelity videos with synchronized body gestures. TANGO builds on Gesture Video Reenactment (GVR), which splits and retrieves video clips using a directed graph structure - representing video frames as nodes and valid transitions as edges. We address two key limitations of GVR: audio-motion misalignment and visual artifacts in GAN-generated transition frames. In particular, (i) we propose retrieving gestures using latent feature distance to improve cross-modal alignment. To ensure the latent features could effectively model the relationship between speech audio and gesture motion, we implement a hierarchical joint embedding space (AuMoCLIP); (ii) we introduce the diffusion-based model to generate high-quality transition frames. Our diffusion model, Appearance Consistent Interpolation (ACInterp), is built upon AnimateAnyone and includes a reference motion module and homography background flow to preserve appearance consistency between generated and reference videos. By integrating these components into the graph-based retrieval framework, TANGO reliably produces realistic, audio-synchronized videos and outperforms all existing generative and retrieval methods. Our codes and pretrained models are available: \url{https://pantomatrix.github.io/TANGO/}