Learning Speaker-Invariant Visual Features for Lipreading

TL;DR

This paper introduces SIFLip, a novel framework for lipreading that learns speaker-invariant visual features by disentangling speaker-specific attributes, thereby improving cross-speaker generalization and accuracy.

Contribution

SIFLip employs implicit and explicit disentanglement modules to effectively decouple speaker-specific features from visual lip representations, enhancing lipreading performance.

Findings

Outperforms state-of-the-art methods on multiple datasets

Significantly improves cross-speaker generalization

Effectively disentangles speaker-specific attributes

Abstract

Lipreading is a challenging cross-modal task that aims to convert visual lip movements into spoken text. Existing lipreading methods often extract visual features that include speaker-specific lip attributes (e.g., shape, color, texture), which introduce spurious correlations between vision and text. These correlations lead to suboptimal lipreading accuracy and restrict model generalization. To address this challenge, we introduce SIFLip, a speaker-invariant visual feature learning framework that disentangles speaker-specific attributes using two complementary disentanglement modules (Implicit Disentanglement and Explicit Disentanglement) to improve generalization. Specifically, since different speakers exhibit semantic consistency between lip movements and phonetic text when pronouncing the same words, our implicit disentanglement module leverages stable text embeddings as supervisory signals to learn common visual representations across speakers, implicitly decoupling speaker-specific features. Additionally, we design a speaker recognition sub-task within the main lipreading pipeline to filter speaker-specific features, then further explicitly disentangle these personalized visual features from the backbone network via gradient reversal. Experimental results demonstrate that SIFLip significantly enhances generalization performance across multiple public datasets. Experimental results demonstrate that SIFLip significantly improves generalization performance across multiple public datasets, outperforming state-of-the-art methods.