Video Editing for Audio-Visual Dubbing

TL;DR

EdiDub is a novel content-aware video editing framework that improves audio-visual dubbing by accurately synchronizing facial movements while preserving original scene details, outperforming existing methods.

Contribution

Introduces EdiDub, a content-aware editing approach for visual dubbing that maintains scene integrity and enhances synchronization accuracy.

Findings

Significantly improves identity preservation and lip synchronization.

Outperforms existing methods in benchmarks, especially with occlusions.

Human evaluations favor EdiDub's naturalness and synchronization.

Abstract

Visual dubbing, the synchronization of facial movements with new speech, is crucial for making content accessible across different languages, enabling broader global reach. However, current methods face significant limitations. Existing approaches often generate talking faces, hindering seamless integration into original scenes, or employ inpainting techniques that discard vital visual information like partial occlusions and lighting variations. This work introduces EdiDub, a novel framework that reformulates visual dubbing as a content-aware editing task. EdiDub preserves the original video context by utilizing a specialized conditioning scheme to ensure faithful and accurate modifications rather than mere copying. On multiple benchmarks, including a challenging occluded-lip dataset, EdiDub significantly improves identity preservation and synchronization. Human evaluations further confirm its superiority, achieving higher synchronization and visual naturalness scores compared to the leading methods. These results demonstrate that our content-aware editing approach outperforms traditional generation or inpainting, particularly in maintaining complex visual elements while ensuring accurate lip synchronization.