Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?

TL;DR

This paper systematically analyzes how multiple modalities, including audio, images, and lip movements, affect speech recognition accuracy, revealing that combining modalities generally improves performance, especially with relevant visual filtering and at moderate noise levels.

Contribution

It is the first to demonstrate the benefits of integrating audio, image context, and lip information in speech recognition models and analyzes their effects under various noise conditions.

Findings

Integrating more modalities increases recognition accuracy.

Images provide significant benefits at moderate noise levels.

Filtering relevant visual information enhances model performance.

Abstract

Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities impact performance in specific scenarios remain limited. In this paper, we investigate the effects of multiple modalities on recognition accuracy on both synthetic and real-world datasets. Our experiments suggest that: (1) Integrating more modalities can increase accuracy; in particular, our paper is, to our best knowledge, the first to show the benefit of combining audio, image context, and lip information; (2) Images as a supplementary modality for speech recognition provide the greatest benefit at moderate noise levels, moreover, they exhibit a different trend compared to inherently synchronized modalities like lip movements; (3) Performance improves on both synthetic and real-world datasets when the most relevant visual information is filtered as a preprocessing step.