Improving Multimodal Speech Recognition by Data Augmentation and Speech Representations

TL;DR

This paper enhances multimodal speech recognition by leveraging pretrained speech models and data augmentation techniques to better integrate visual information, resulting in improved performance across multiple datasets.

Contribution

It introduces the use of pretrained ASR models and novel speech data augmentation methods to improve multimodal speech recognition performance.

Findings

Pretrained ASR models significantly boost performance.

Speech data augmentation improves multimodal attention to visual stimuli.

Consistent gains observed across three datasets, including Localized Narratives.

Abstract

Multimodal speech recognition aims to improve the performance of automatic speech recognition (ASR) systems by leveraging additional visual information that is usually associated to the audio input. While previous approaches make crucial use of strong visual representations, e.g. by finetuning pretrained image recognition networks, significantly less attention has been paid to its counterpart: the speech component. In this work, we investigate ways of improving the base speech recognition system by following similar techniques to the ones used for the visual encoder, namely, transferring representations and data augmentation. First, we show that starting from a pretrained ASR significantly improves the state-of-the-art performance; remarkably, even when building upon a strong unimodal system, we still find gains by including the visual modality. Second, we employ speech data augmentation techniques to encourage the multimodal system to attend to the visual stimuli. This technique replaces previously used word masking and comes with the benefits of being conceptually simpler and yielding consistent improvements in the multimodal setting. We provide empirical results on three multimodal datasets, including the newly introduced Localized Narratives.