TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model

TL;DR

TalkingPose is a diffusion-based framework that generates long-form, temporally coherent face and gesture animations from a single image, using feedback mechanisms to improve continuity without extra training.

Contribution

It introduces a feedback-guided diffusion model for long-term, coherent human upper-body animation, overcoming computational limits of previous short-segment methods.

Findings

Enables unlimited-duration, coherent animations

Uses feedback mechanism without extra training or costs

Provides a new large-scale benchmark dataset

Abstract

Recent advancements in diffusion models have significantly improved the realism and generalizability of character-driven animation, enabling the synthesis of high-quality motion from just a single RGB image and a set of driving poses. Nevertheless, generating temporally coherent long-form content remains challenging. Existing approaches are constrained by computational and memory limitations, as they are typically trained on short video segments, thus performing effectively only over limited frame lengths and hindering their potential for extended coherent generation. To address these constraints, we propose TalkingPose, a novel diffusion-based framework specifically designed for producing long-form, temporally consistent human upper-body animations. TalkingPose leverages driving frames to precisely capture expressive facial and hand movements, transferring these seamlessly to a target actor through a stable diffusion backbone. To ensure continuous motion and enhance temporal coherence, we introduce a feedback-driven mechanism built upon image-based diffusion models. Notably, this mechanism does not incur additional computational costs or require secondary training stages, enabling the generation of animations with unlimited duration. Additionally, we introduce a comprehensive, large-scale dataset to serve as a new benchmark for human upper-body animation.