MLLM-based Speech Recognition: When and How is Multimodality Beneficial?

TL;DR

This paper investigates when and how multimodal inputs, such as lip movements and images, enhance speech recognition accuracy, especially in noisy environments, revealing that modality type, synchronization, quality, and model architecture influence benefits.

Contribution

It systematically analyzes the conditions and model configurations under which multimodal data improves speech recognition performance in noisy settings.

Findings

Multimodal inputs generally improve ASR accuracy, especially with more noise.

Synchronized modalities are more beneficial at high noise levels.

High-quality visual representations significantly boost ASR performance.

Abstract

Recent advances in multi-modal large language models (MLLMs) have opened new possibilities for unified modeling of speech, text, images, and other modalities. Building on our prior work, this paper examines the conditions and model architectures under which multiple input modalities can improve automatic speech recognition (ASR) accuracy in noisy environments. Through experiments on synthetic and real-world data, we find that (1) harnessing more modalities usually improves ASR accuracy, as each modality provides complementary information, but the improvement depends on the amount of auditory noise. (2) Synchronized modalities (e.g., lip movements) are more useful at high noise levels whereas unsynchronized modalities (e.g., image context) are most helpful at moderate noise levels. (3) Higher-quality visual representations consistently improve ASR accuracy, highlighting the importance of developing more powerful visual encoders. (4) Mamba exhibits similar trends regarding the benefits of multimodality as do Transformers. (5) The input order of modalities as well as their weights in the loss function can significantly impact accuracy. These findings both offer practical insights and help to deepen our understanding of multi-modal speech recognition under challenging conditions.