End-to-End Multimodal Speech Recognition

TL;DR

This paper explores end-to-end multimodal speech recognition models that incorporate visual context to improve transcription accuracy in challenging open-domain videos, analyzing robustness across noisy and clean data.

Contribution

It introduces an end-to-end multimodal approach combining visual features with speech recognition, comparing CTC and sequence-to-sequence models on noisy and clean datasets.

Findings

Visual features improve recognition accuracy in noisy videos

Sequence-to-sequence models adapt both acoustic and language information jointly

CTC models show robustness in noisy conditions

Abstract

Transcription or sub-titling of open-domain videos is still a challenging domain for Automatic Speech Recognition (ASR) due to the data's challenging acoustics, variable signal processing and the essentially unrestricted domain of the data. In previous work, we have shown that the visual channel -- specifically object and scene features -- can help to adapt the acoustic model (AM) and language model (LM) of a recognizer, and we are now expanding this work to end-to-end approaches. In the case of a Connectionist Temporal Classification (CTC)-based approach, we retain the separation of AM and LM, while for a sequence-to-sequence (S2S) approach, both information sources are adapted together, in a single model. This paper also analyzes the behavior of CTC and S2S models on noisy video data (How-To corpus), and compares it to results on the clean Wall Street Journal (WSJ) corpus, providing insight into the robustness of both approaches.