EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers

TL;DR

EditYourself is a diffusion transformer-based framework that enables precise, audio-driven editing of talking head videos, allowing seamless content modification while preserving motion, identity, and lip sync.

Contribution

It introduces a novel audio-conditioned video editing method using diffusion transformers, enabling transcript-based modifications with high fidelity and temporal coherence.

Findings

Achieves accurate lip synchronization in edited videos.

Maintains visual identity and motion coherence over long durations.

Enables realistic addition, removal, and retiming of speech segments.

Abstract

Current generative video models excel at producing novel content from text and image prompts, but leave a critical gap in editing existing pre-recorded videos, where minor alterations to the spoken script require preserving motion, temporal coherence, speaker identity, and accurate lip synchronization. We introduce EditYourself, a DiT-based framework for audio-driven video-to-video (V2V) editing that enables transcript-based modification of talking head videos, including the seamless addition, removal, and retiming of visually spoken content. Building on a general-purpose video diffusion model, EditYourself augments its V2V capabilities with audio conditioning and region-aware, edit-focused training extensions. This enables precise lip synchronization and temporally coherent restructuring of existing performances via spatiotemporal inpainting, including the synthesis of realistic human motion in newly added segments, while maintaining visual fidelity and identity consistency over long durations. This work represents a foundational step toward generative video models as practical tools for professional video post-production.