Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation

TL;DR

This paper introduces a comprehensive approach combining cross-lingual transfer learning, speaker adaptation, and model ensembling to improve lipreading accuracy across languages and speakers, achieving state-of-the-art results in Chinese lipreading.

Contribution

It proposes novel methods for cross-lingual transfer, speaker adaptation, and model ensembling to enhance lipreading performance in multilingual and speaker-specific contexts.

Findings

Achieved CER of 77.3% on ChatCLR dataset

Outperformed top results in 2024 Chat-scenario Chinese Lipreading Challenge

Demonstrated effectiveness of combining lip ROI and face inputs

Abstract

Lipreading is an important technique for facilitating human-computer interaction in noisy environments. Our previously developed self-supervised learning method, AV2vec, which leverages multimodal self-distillation, has demonstrated promising performance in speaker-independent lipreading on the English LRS3 dataset. However, AV2vec faces challenges such as high training costs and a potential scarcity of audio-visual data for lipreading in languages other than English, such as Chinese. Additionally, most studies concentrate on speakerindependent lipreading models, which struggle to account for the substantial variation in speaking styles across di?erent speakers. To address these issues, we propose a comprehensive approach. First, we investigate cross-lingual transfer learning, adapting a pre-trained AV2vec model from a source language and optimizing it for the lipreading task in a target language. Second, we enhance the accuracy of lipreading for specific target speakers through a speaker adaptation strategy, which is not extensively explored in previous research. Third, after analyzing the complementary performance of lipreading with lip region-of-interest (ROI) and face inputs, we introduce a model ensembling strategy that integrates both, signi?cantly boosting model performance. Our method achieved a character error rate (CER) of 77.3% on the evaluation set of the ChatCLR dataset, which is lower than the top result from the 2024 Chat-scenario Chinese Lipreading Challenge.