Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization

TL;DR

This paper introduces a landmark-guided lip reading method with mutual information regularization to improve cross-speaker robustness by reducing speaker-specific visual variations and capturing speaker-invariant features.

Contribution

It proposes using lip landmarks as input features and a mutual information regularization to enhance speaker-robust lip reading models, addressing inter-speaker variability.

Findings

Improved accuracy in cross-speaker lip reading tasks.

Effective reduction of speaker-specific appearance influence.

Enhanced model generalization across different speakers.

Abstract

Lip reading, the process of interpreting silent speech from visual lip movements, has gained rising attention for its wide range of realistic applications. Deep learning approaches greatly improve current lip reading systems. However, lip reading in cross-speaker scenarios where the speaker identity changes, poses a challenging problem due to inter-speaker variability. A well-trained lip reading system may perform poorly when handling a brand new speaker. To learn a speaker-robust lip reading model, a key insight is to reduce visual variations across speakers, avoiding the model overfitting to specific speakers. In this work, in view of both input visual clues and latent representations based on a hybrid CTC/attention architecture, we propose to exploit the lip landmark-guided fine-grained visual clues instead of frequently-used mouth-cropped images as input features, diminishing speaker-specific appearance characteristics. Furthermore, a max-min mutual information regularization approach is proposed to capture speaker-insensitive latent representations. Experimental evaluations on public lip reading datasets demonstrate the effectiveness of the proposed approach under the intra-speaker and inter-speaker conditions.