Controllable Talking Face Generation by Implicit Facial Keypoints Editing

TL;DR

ControlTalk is a novel method for controllable talking face generation that achieves natural lip synchronization and expression control across diverse inputs and identities, simplifying editing processes in digital human applications.

Contribution

It introduces a unified framework with lightweight adaptation for precise expression and lip motion control, outperforming state-of-the-art methods on standard benchmarks.

Findings

Superior performance on HDTF and MEAD benchmarks.

Effective generalization across same-ID, cross-ID, and out-of-domain portraits.

Accurate lip synchronization with quantitative mouth opening control.

Abstract

Audio-driven talking face generation has garnered significant interest within the domain of digital human research. Existing methods are encumbered by intricate model architectures that are intricately dependent on each other, complicating the process of re-editing image or video inputs. In this work, we present ControlTalk, a talking face generation method to control face expression deformation based on driven audio, which can construct the head pose and facial expression including lip motion for both single image or sequential video inputs in a unified manner. By utilizing a pre-trained video synthesis renderer and proposing the lightweight adaptation, ControlTalk achieves precise and naturalistic lip synchronization while enabling quantitative control over mouth opening shape. Our experiments show that our method is superior to state-of-the-art performance on widely used benchmarks, including HDTF and MEAD. The parameterized adaptation demonstrates remarkable generalization capabilities, effectively handling expression deformation across same-ID and cross-ID scenarios, and extending its utility to out-of-domain portraits, regardless of languages. Code is available at https://github.com/NetEase-Media/ControlTalk.