LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation

TL;DR

LawDNet introduces a novel deep-learning approach with local affine warping to improve the realism, diversity, and temporal coherence of audio-driven lip synthesis for photorealistic avatars.

Contribution

The paper presents LawDNet, a new architecture that models lip movements using local affine warping fields and a dual-stream discriminator, enhancing lip synthesis quality and robustness.

Findings

Outperforms previous methods in lip movement realism and diversity

Achieves superior temporal coherence and robustness in lip synthesis

Provides accessible source code and pre-trained models for research community

Abstract

In the domain of photorealistic avatar generation, the fidelity of audio-driven lip motion synthesis is essential for realistic virtual interactions. Existing methods face two key challenges: a lack of vivacity due to limited diversity in generated lip poses and noticeable anamorphose motions caused by poor temporal coherence. To address these issues, we propose LawDNet, a novel deep-learning architecture enhancing lip synthesis through a Local Affine Warping Deformation mechanism. This mechanism models the intricate lip movements in response to the audio input by controllable non-linear warping fields. These fields consist of local affine transformations focused on abstract keypoints within deep feature maps, offering a novel universal paradigm for feature warping in networks. Additionally, LawDNet incorporates a dual-stream discriminator for improved frame-to-frame continuity and employs face normalization techniques to handle pose and scene variations. Extensive evaluations demonstrate LawDNet's superior robustness and lip movement dynamism performance compared to previous methods. The advancements presented in this paper, including the methodologies, training data, source codes, and pre-trained models, will be made accessible to the research community.