OPT: One-shot Pose-Controllable Talking Head Generation

TL;DR

This paper introduces OPT, a novel one-shot talking head generation method that achieves high-quality, pose-controllable lip-sync videos while preserving source identity, overcoming previous identity mismatch issues.

Contribution

OPT is the first method to effectively disentangle audio content from speaker identity and incorporate pose control, enabling realistic, identity-preserving talking head synthesis.

Findings

Outperforms previous state-of-the-art methods in quality and pose control.

Successfully preserves source identity during pose variations.

Generates natural and lip-synced talking heads with high fidelity.

Abstract

One-shot talking head generation produces lip-sync talking heads based on arbitrary audio and one source face. To guarantee the naturalness and realness, recent methods propose to achieve free pose control instead of simply editing mouth areas. However, existing methods do not preserve accurate identity of source face when generating head motions. To solve the identity mismatch problem and achieve high-quality free pose control, we present One-shot Pose-controllable Talking head generation network (OPT). Specifically, the Audio Feature Disentanglement Module separates content features from audios, eliminating the influence of speaker-specific information contained in arbitrary driving audios. Later, the mouth expression feature is extracted from the content feature and source face, during which the landmark loss is designed to enhance the accuracy of facial structure and identity preserving quality. Finally, to achieve free pose control, controllable head pose features from reference videos are fed into the Video Generator along with the expression feature and source face to generate new talking heads. Extensive quantitative and qualitative experimental results verify that OPT generates high-quality pose-controllable talking heads with no identity mismatch problem, outperforming previous SOTA methods.