When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse

TL;DR

This paper evaluates the robustness of AVSR models in video conferencing, constructs a new dataset with Lombard effect, and uncovers how speech enhancement causes distribution shifts affecting model performance.

Contribution

It introduces MLD-VC, the first multimodal dataset for VC, and analyzes the impact of speech enhancement and Lombard effect on AVSR robustness.

Findings

Speech enhancement algorithms cause distribution shifts in audio features.

Lombard effect-induced shifts resemble those from speech enhancement, improving robustness.

Fine-tuning on MLD-VC reduces CER by 17.5% on average across VC platforms.

Abstract

Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic evaluation of state-of-the-art AVSR models across mainstream VC platforms, revealing severe performance degradation caused by transmission distortions and spontaneous human hyper-expression. To address this gap, we construct \textbf{MLD-VC}, the first multimodal dataset tailored for VC, comprising 31 speakers, 22.79 hours of audio-visual data, and explicit use of the Lombard effect to enhance human hyper-expression. Through comprehensive analysis, we find that speech enhancement algorithms are the primary source of distribution shift, which alters the first and second formants of audio. Interestingly, we find that the distribution shift induced by the Lombard effect closely resembles that introduced by speech enhancement, which explains why models trained on Lombard data exhibit greater robustness in VC. Fine-tuning AVSR models on MLD-VC mitigates this issue, achieving an average 17.5% reduction in CER across several VC platforms. Our findings and dataset provide a foundation for developing more robust and generalizable AVSR systems in real-world video conferencing. MLD-VC is available at https://huggingface.co/datasets/nccm2p2/MLD-VC.