Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

TL;DR

This paper presents a unified system that simultaneously generates realistic talking faces and speech from text, addressing pose diversity and voice consistency challenges with novel motion sampling and conditioning techniques.

Contribution

It introduces a joint framework combining talking face generation and TTS, with innovative motion sampling and conditioning methods for improved multimodal synthesis.

Findings

Produces natural-looking talking faces and speech matching input text

Effectively handles diverse head poses and voice consistency

Generalizes to unseen identities in multimodal synthesis

Abstract

The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities.