Show Me Your Face, And I'll Tell You How You Speak

TL;DR

This paper introduces Lip2Speech, a novel lip-to-speech synthesis method that generates speaker-specific speech from lip movements across multiple speakers in unconstrained, large vocabulary scenarios.

Contribution

The work presents a new approach for speaker-aware lip-to-speech synthesis that captures facial features to improve speech generation accuracy in diverse, real-world conditions.

Findings

Lip2Speech achieves high accuracy in lip-to-speech mapping.

The method effectively captures speaker identity features.

Extensive evaluations demonstrate its robustness and realism.

Abstract

When we speak, the prosody and content of the speech can be inferred from the movement of our lips. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate speech given only the lip movements of a speaker where we focus on learning accurate lip to speech mappings for multiple speakers in unconstrained, large vocabulary settings. We capture the speaker's voice identity through their facial characteristics, i.e., age, gender, ethnicity and condition them along with the lip movements to generate speaker identity aware speech. To this end, we present a novel method "Lip2Speech", with key design choices to achieve accurate lip to speech synthesis in unconstrained scenarios. We also perform various experiments and extensive evaluation using quantitative, qualitative metrics and human evaluation.