HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis

TL;DR

HM-Talker introduces a hybrid motion modeling framework combining implicit and explicit cues, including anatomical facial muscle movements, to generate high-fidelity, temporally coherent talking head videos with improved lip-sync and visual quality.

Contribution

The paper presents HM-Talker, a novel framework that explicitly incorporates Action Units for better lip synchronization and cross-subject generalization in talking head synthesis.

Findings

Outperforms state-of-the-art in visual quality

Achieves superior lip-sync accuracy

Enhances cross-subject generalization

Abstract

Audio-driven talking head video generation enhances user engagement in human-computer interaction. However, current methods frequently produce videos with motion blur and lip jitter, primarily due to their reliance on implicit modeling of audio-facial motion correlations--an approach lacking explicit articulatory priors (i.e., anatomical guidance for speech-related facial movements). To overcome this limitation, we propose HM-Talker, a novel framework for generating high-fidelity, temporally coherent talking heads. HM-Talker leverages a hybrid motion representation combining both implicit and explicit motion cues. Explicit cues use Action Units (AUs), anatomically defined facial muscle movements, alongside implicit features to minimize phoneme-viseme misalignment. Specifically, our Cross-Modal Disentanglement Module (CMDM) extracts complementary implicit/explicit motion features while predicting AUs directly from audio input aligned to visual cues. To mitigate identity-dependent biases in explicit features and enhance cross-subject generalization, we introduce the Hybrid Motion Modeling Module (HMMM). This module dynamically merges randomly paired implicit/explicit features, enforcing identity-agnostic learning. Together, these components enable robust lip synchronization across diverse identities, advancing personalized talking head synthesis. Extensive experiments demonstrate HM-Talker's superiority over state-of-the-art methods in visual quality and lip-sync accuracy.