VisG AV-HuBERT: Viseme-Guided AV-HuBERT

TL;DR

VisG AV-HuBERT introduces a multi-task framework incorporating viseme classification to improve audiovisual speech recognition, especially under noisy conditions, by explicitly guiding the encoder to focus on visual speech features.

Contribution

The paper presents a novel viseme-guided fine-tuning approach for AV-HuBERT that enhances visual feature encoding and noise robustness in AVSR systems.

Findings

Achieves 51.4% relative WER reduction at -10 dB SNR compared to baseline.

Improves speech unit discrimination and reduces substitution errors across noise types.

Demonstrates generalization on LRS2 dataset.

Abstract

Audio-Visual Speech Recognition (AVSR) systems nowadays integrate Large Language Model (LLM) decoders with transformer-based encoders, achieving state-of-the-art results. However, the relative contributions of improved language modelling versus enhanced audiovisual encoding remain unclear. We propose Viseme-Guided AV-HuBERT (VisG AV-HuBERT), a multi-task fine-tuning framework that incorporates auxiliary viseme classification to strengthen the model's reliance on visual articulatory features. By extending AV-HuBERT with a lightweight viseme prediction sub-network, this method explicitly guides the encoder to preserve visual speech information. Evaluated on LRS3, VisG AV-HuBERT achieves comparable or improved performance over the baseline AV-HuBERT, with notable gains under heavy noise conditions. WER reduces from 13.59% to 6.60% (51.4% relative improvement) at -10 dB Signal-to-Noise Ratio (SNR) for Speech noise. Deeper analysis reveals substantial reductions in substitution errors across noise types, demonstrating improved speech unit discrimination. Evaluation on LRS2 confirms generalization capability. Our results demonstrate that explicit viseme modelling enhances encoder representations, and provides a foundation for enhancing noise-robust AVSR through encoder-level improvements.