Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis

TL;DR

This paper introduces a novel text-to-talking-face synthesis method that uses shared latent speech representations to generate synchronized speech and facial animations, improving realism and lip-sync accuracy.

Contribution

It presents a joint text-to-audio-visual synthesis framework leveraging latent speech features and a two-stage training process to enhance alignment and realism.

Findings

Outperforms cascaded pipelines in lip-sync accuracy

Produces more natural and expressive speech and facial animations

Maintains speaker identity without ground-truth audio at inference

Abstract

We propose a text-to-talking-face synthesis framework leveraging latent speech representations from HierSpeech++. A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation. To handle distribution shifts between clean and TTS-predicted features, we adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs. This enables tight audio-visual alignment, preserves speaker identity, and produces natural, expressive speech and synchronized facial motion without ground-truth audio at inference. Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines, improving both lip-sync and visual realism.