Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation

TL;DR

This paper introduces THFEM, a framework combining audio-driven talking head models with SPFEM, using adjacent frame learning to improve lip synchronization and expression fidelity in facial manipulation tasks.

Contribution

The paper proposes a novel integration of AD-THG models with SPFEM and introduces an adjacent frame learning strategy to enhance image quality and lip synchronization.

Findings

Improved lip synchronization in manipulated images.

Enhanced realism and expression fidelity through adjacent frame learning.

Effective preservation of mouth shapes during facial expression manipulation.

Abstract

Speech-Preserving Facial Expression Manipulation (SPFEM) is an innovative technique aimed at altering facial expressions in images and videos while retaining the original mouth movements. Despite advancements, SPFEM still struggles with accurate lip synchronization due to the complex interplay between facial expressions and mouth shapes. Capitalizing on the advanced capabilities of audio-driven talking head generation (AD-THG) models in synthesizing precise lip movements, our research introduces a novel integration of these models with SPFEM. We present a new framework, Talking Head Facial Expression Manipulation (THFEM), which utilizes AD-THG models to generate frames with accurately synchronized lip movements from audio inputs and SPFEM-altered images. However, increasing the number of frames generated by AD-THG models tends to compromise the realism and expression fidelity of the images. To counter this, we develop an adjacent frame learning strategy that finetunes AD-THG models to predict sequences of consecutive frames. This strategy enables the models to incorporate information from neighboring frames, significantly improving image quality during testing. Our extensive experimental evaluations demonstrate that this framework effectively preserves mouth shapes during expression manipulations, highlighting the substantial benefits of integrating AD-THG with SPFEM.