Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR

TL;DR

This paper introduces a new multimodal ASR task for scientific videos, proposing a severity-aware WER metric and a vision-augmented framework that significantly improves transcription accuracy by leveraging visual information.

Contribution

It presents the MS-ASR task, a severity-aware WER metric, and the SciVASR framework, advancing multimodal ASR for scientific content with substantial performance gains.

Findings

45% improvement over speech-only baselines

Severity-aware WER better evaluates error severity

Multimodal integration enhances transcript quality

Abstract

Recent advancements in multimodal large language models (MLLMs) have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. This paper introduces the Multimodal Scientific ASR (MS-ASR) task, which focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. Realized that traditional metrics like WER fall short in assessing performance accurately, prompting the proposal of severity-aware WER (SWER) that considers the content type and severity of ASR errors. We propose the Scientific Vision Augmented ASR (SciVASR) framework as a baseline method, enabling MLLMs to improve transcript quality through post-editing. Evaluations of state-of-the-art MLLMs, including GPT-4o, show a 45% improvement over speech-only baselines, highlighting the importance of multimodal information integration.