Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language

TL;DR

This paper introduces a novel speaker-adaptive lip reading method that adapts to individual speakers at both visual and language levels, utilizing prompt tuning and LoRA, validated on a new diverse dataset for real-world sentence-level lip reading.

Contribution

It proposes a new speaker adaptation approach combining prompt tuning and LoRA, and introduces VoxLRS-SA, a large, diverse dataset for real-world, sentence-level lip reading.

Findings

Existing methods improve with speaker adaptation in the wild.

The proposed method outperforms previous approaches.

VoxLRS-SA enables validation of lip reading in diverse real-world scenarios.

Abstract

Lip reading aims to predict spoken language by analyzing lip movements. Despite advancements in lip reading technologies, performance degrades when models are applied to unseen speakers due to their sensitivity to variations in visual information such as lip appearances. To address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality. However, the effectiveness of adapting language information, such as vocabulary choice, of the target speaker has not been explored in previous works. Additionally, existing datasets for speaker adaptation have limited vocabulary sizes and pose variations, which restrict the validation of previous speaker-adaptive methods in real-world scenarios. To address these issues, we propose a novel speaker-adaptive lip reading method that adapts a pre-trained model to target speakers at both vision and language levels. Specifically, we integrate prompt tuning and the LoRA approach, applying them to a pre-trained lip reading model to effectively adapt the model to target speakers. Furthermore, to validate its effectiveness in real-world scenarios, we introduce a new dataset, VoxLRS-SA, derived from VoxCeleb2 and LRS3. It contains a vocabulary of approximately 100K words, offers diverse pose variations, and enables the validation of adaptation methods in the wild, sentence-level lip reading for the first time in English. Through various experiments, we demonstrate that the existing speaker-adaptive method also improves performance in the wild at the sentence level. Moreover, we show that the proposed method achieves larger improvements compared to the previous works.