Deep Multimodal Learning for Audio-Visual Speech Recognition

TL;DR

This paper introduces deep multimodal learning methods for audio-visual speech recognition, demonstrating improved performance by fusing speech and visual data with novel architectures and fusion strategies.

Contribution

It proposes a new deep network architecture with bilinear softmax for modality correlation and demonstrates enhanced AV-ASR performance through multimodal fusion.

Findings

Fused models reduce PER from 41% to 35.83% in clean conditions.

Bilinear softmax architecture captures class-specific modality correlations.

Combining models achieves a PER of 34.03%, outperforming individual approaches.

Abstract

In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of $41\%$ under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of $35.83\%$ demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of $34.03\%$.