# An Analysis of Speech Enhancement and Recognition Losses in Limited Resources Multi-talker Single Channel Audio-Visual ASR

## TL;DR

This paper investigates how joint training of speech enhancement and recognition models impacts performance in multi-talker audio-visual scenarios, revealing complex interactions and improvements in phone error rates.

## Contribution

It introduces and analyzes joint end-to-end models for speech enhancement and recognition in multi-talker settings, highlighting training strategies and their effects.

## Key findings

- Joint optimization reduces Phone Error Rate significantly.
- Speech enhancement capability decreases during ASR training.
- Models trained jointly outperform baseline models trained separately.

## Abstract

In this paper, we analyzed how audio-visual speech enhancement can help to perform the ASR task in a cocktail party scenario. Therefore we considered two simple end-to-end LSTM-based models that perform single-channel audio-visual speech enhancement and phone recognition respectively. Then, we studied how the two models interact, and how to train them jointly affects the final result. We analyzed different training strategies that reveal some interesting and unexpected behaviors. The experiments show that during optimization of the ASR task the speech enhancement capability of the model significantly decreases and vice-versa. Nevertheless the joint optimization of the two tasks shows a remarkable drop of the Phone Error Rate (PER) compared to the audio-visual baseline models trained only to perform phone recognition. We analyzed the behaviors of the proposed models by using two limited-size datasets, and in particular we used the mixed-speech versions of GRID and TCD-TIMIT.