Parallel and High-Fidelity Text-to-Lip Generation

TL;DR

This paper introduces ParaLip, a parallel decoding model for text-to-lip generation that achieves faster inference and higher quality lip movements by predicting durations and using non-autoregressive modeling, enhanced with perceptual loss and adversarial training.

Contribution

The work presents a novel parallel decoding approach for T2L generation that improves inference speed and lip quality over autoregressive methods.

Findings

Outperforms existing methods on GRID and TCD-TIMIT datasets.

Achieves faster inference with high-fidelity lip synthesis.

Uses structural similarity and adversarial loss for perceptual enhancement.

Abstract

As a key component of talking face generation, lip movements generation determines the naturalness and coherence of the generated talking face video. Prior literature mainly focuses on speech-to-lip generation while there is a paucity in text-to-lip (T2L) generation. T2L is a challenging task and existing end-to-end works depend on the attention mechanism and autoregressive (AR) decoding manner. However, the AR decoding manner generates current lip frame conditioned on frames generated previously, which inherently hinders the inference speed, and also has a detrimental effect on the quality of generated lip frames due to error propagation. This encourages the research of parallel T2L generation. In this work, we propose a parallel decoding model for fast and high-fidelity text-to-lip generation (ParaLip). Specifically, we predict the duration of the encoded linguistic features and model the target lip frames conditioned on the encoded linguistic features with their duration in a non-autoregressive manner. Furthermore, we incorporate the structural similarity index loss and adversarial learning to improve perceptual quality of generated lip frames and alleviate the blurry prediction problem. Extensive experiments conducted on GRID and TCD-TIMIT datasets demonstrate the superiority of proposed methods. Video samples are available via \url{https://paralip.github.io/}.