Identity-Preserving Video Dubbing Using Motion Warping

TL;DR

This paper introduces IPTalker, a transformer-based framework for video dubbing that achieves high-fidelity identity preservation and lip-sync accuracy by dynamically aligning audio cues with reference visuals and refining the generated videos.

Contribution

The paper presents a novel transformer-based alignment mechanism combined with motion warping and refinement strategies to improve identity preservation in video dubbing.

Findings

Outperforms existing methods in realism and lip-sync accuracy

Achieves superior identity retention in generated videos

Establishes new state-of-the-art in identity-consistent video dubbing

Abstract

Video dubbing aims to synthesize realistic, lip-synced videos from a reference video and a driving audio signal. Although existing methods can accurately generate mouth shapes driven by audio, they often fail to preserve identity-specific features, largely because they do not effectively capture the nuanced interplay between audio cues and the visual attributes of reference identity . As a result, the generated outputs frequently lack fidelity in reproducing the unique textural and structural details of the reference identity. To address these limitations, we propose IPTalker, a novel and robust framework for video dubbing that achieves seamless alignment between driving audio and reference identity while ensuring both lip-sync accuracy and high-fidelity identity preservation. At the core of IPTalker is a transformer-based alignment mechanism designed to dynamically capture and model the correspondence between audio features and reference images, thereby enabling precise, identity-aware audio-visual integration. Building on this alignment, a motion warping strategy further refines the results by spatially deforming reference images to match the target audio-driven configuration. A dedicated refinement process then mitigates occlusion artifacts and enhances the preservation of fine-grained textures, such as mouth details and skin features. Extensive qualitative and quantitative evaluations demonstrate that IPTalker consistently outperforms existing approaches in terms of realism, lip synchronization, and identity retention, establishing a new state of the art for high-quality, identity-consistent video dubbing.