Speech Recognition on TV Series with Video-guided Post-ASR Correction

TL;DR

This paper introduces a Video-Guided Post-ASR Correction framework that leverages video context through a Video-Large Multimodal Model to improve transcription accuracy in complex TV series environments.

Contribution

It presents a novel multimodal correction approach that explicitly utilizes video information to enhance ASR transcription quality in challenging multimedia settings.

Findings

Significant improvement in transcription accuracy on TV-series benchmark.

Effective utilization of video context for post-ASR correction.

Demonstrated robustness in complex multimedia environments.

Abstract

Automatic Speech Recognition (ASR) has achieved remarkable success with deep learning, driving advancements in conversational artificial intelligence, media transcription, and assistive technologies. However, ASR systems still struggle in complex environments such as TV series, where multiple speakers, overlapping speech, domain-specific terminology, and long-range contextual dependencies pose significant challenges to transcription accuracy. Existing approaches fail to explicitly leverage the rich temporal and contextual information available in the video. To address this limitation, we propose a Video-Guided Post-ASR Correction (VPC) framework that uses a Video-Large Multimodal Model (VLMM) to capture video context and refine ASR outputs. Evaluations on a TV-series benchmark show that our method consistently improves transcription accuracy in complex multimedia environments.