Intelligent Video Editing: Incorporating Modern Talking Face Generation Algorithms in a Video Editor

TL;DR

This paper introduces an advanced video editing tool that integrates modern talking face generation algorithms, enabling interactive lip-syncing, facial re-enactment, and automatic translation features to enhance synthetic video creation.

Contribution

It presents a user-friendly video editor that combines state-of-the-art facial video editing algorithms with manual controls for improved editing and synthesis quality.

Findings

Human evaluations show increased editing efficiency.

Improved quality of synthetic talking face videos.

Effective synchronization of background content with translated speech.

Abstract

This paper proposes a video editor based on OpenShot with several state-of-the-art facial video editing algorithms as added functionalities. Our editor provides an easy-to-use interface to apply modern lip-syncing algorithms interactively. Apart from lip-syncing, the editor also uses audio and facial re-enactment to generate expressive talking faces. The manual control improves the overall experience of video editing without missing out on the benefits of modern synthetic video generation algorithms. This control enables us to lip-sync complex dubbed movie scenes, interviews, television shows, and other visual content. Furthermore, our editor provides features that automatically translate lectures from spoken content, lip-sync of the professor, and background content like slides. While doing so, we also tackle the critical aspect of synchronizing background content with the translated speech. We qualitatively evaluate the usefulness of the proposed editor by conducting human evaluations. Our evaluations show a clear improvement in the efficiency of using human editors and an improved video generation quality. We attach demo videos with the supplementary material clearly explaining the tool and also showcasing multiple results.