Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers

TL;DR

This paper introduces AV-CAT, a transformer-based framework that accurately predicts and inpaints mouth shapes for lip-syncing, leveraging audio-visual context to produce realistic talking face videos with minimal facial deformation.

Contribution

The work presents a novel hybrid convolution-Transformer model with an attention-based fusion strategy specifically designed for mouth inpainting in lip-sync generation, focusing on realistic and targeted facial region editing.

Findings

Achieves high-fidelity lip-synced results for arbitrary subjects.

Outperforms existing methods in realism and accuracy.

Demonstrates effective use of audio-visual context in lip movement prediction.

Abstract

Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. However, most of them deform or generate the whole facial area, leading to non-realistic results. In this work, we delve into the formulation of altering only the mouth shapes of the target person. This requires masking a large percentage of the original image and seamlessly inpainting it with the aid of audio and reference frames. To this end, we propose the Audio-Visual Context-Aware Transformer (AV-CAT) framework, which produces accurate lip-sync with photo-realistic quality by predicting the masked mouth shapes. Our key insight is to exploit desired contextual information provided in audio and visual modalities thoroughly with delicately designed Transformers. Specifically, we propose a convolution-Transformer hybrid backbone and design an attention-based fusion strategy for filling the masked parts. It uniformly attends to the textural information on the unmasked regions and the reference frame. Then the semantic audio information is involved in enhancing the self-attention computation. Additionally, a refinement network with audio injection improves both image and lip-sync quality. Extensive experiments validate that our model can generate high-fidelity lip-synced results for arbitrary subjects.