# Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

## TL;DR

This paper introduces a resource-efficient audio-visual speech recognition system combining acoustic and lip-reading features with an attention mechanism, significantly improving phoneme recognition accuracy in noisy environments.

## Contribution

The work presents a novel multi-modal neural network architecture that integrates acoustic and visual data with attention, optimized for resource constraints in noisy speech recognition.

## Key findings

- Achieves 75.70% phoneme accuracy on clean audio
- Achieves 58.55% phoneme accuracy at 0dB SNR with noise
- Outperforms state-of-the-art by over 14 percentage points in noisy conditions

## Abstract

Today's Automatic Speech Recognition systems only rely on acoustic signals and often don't perform well under noisy conditions. Performing multi-modal speech recognition - processing acoustic speech signals and lip-reading video simultaneously - significantly enhances the performance of such systems, especially in noisy environments. This work presents the design of such an audio-visual system for Automated Speech Recognition, taking memory and computation requirements into account. First, a Long-Short-Term-Memory neural network for acoustic speech recognition is designed. Second, Convolutional Neural Networks are used to model lip-reading features. These are combined with an LSTM network to model temporal dependencies and perform automatic lip-reading on video. Finally, acoustic-speech and visual lip-reading networks are combined to process acoustic and visual features simultaneously. An attention mechanism ensures performance of the model in noisy environments. This system is evaluated on the TCD-TIMIT 'lipspeaker' dataset for audio-visual phoneme recognition with clean audio and with additive white noise at an SNR of 0dB. It achieves 75.70% and 58.55% phoneme accuracy respectively, over 14 percentage points better than the state-of-the-art for all noise levels.