Investigations on End-to-End Audiovisual Fusion

TL;DR

This paper introduces an end-to-end neural network architecture for audiovisual speech recognition that improves noise robustness and adapts to varying noise levels without separate decision fusion modeling.

Contribution

It presents a novel end-to-end AVSR neural network that outperforms traditional systems and automatically adapts to different noise conditions.

Findings

Outperforms single-modality recognition under all noise conditions

Automatically adapts to different noise levels

Eliminates need for separate decision fusion modeling

Abstract

Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.