Improving Voice Separation by Incorporating End-to-end Speech Recognition

TL;DR

This paper introduces a novel voice separation method that leverages end-to-end speech recognition features to improve performance in noisy and data-limited scenarios, surpassing existing models including audio-visual approaches.

Contribution

It is the first to incorporate deep features from end-to-end speech recognition into voice separation, enhancing long-term phonetic context understanding.

Findings

Significant improvement in signal-to-distortion ratio over baseline models

Outperforms audio-visual models utilizing lip movement information

Effective in noisy and data-limited environments

Abstract

Despite recent advances in voice separation methods, many challenges remain in realistic scenarios such as noisy recording and the limits of available data. In this work, we propose to explicitly incorporate the phonetic and linguistic nature of speech by taking a transfer learning approach using an end-to-end automatic speech recognition (E2EASR) system. The voice separation is conditioned on deep features extracted from E2EASR to cover the long-term dependence of phonetic aspects. Experimental results on speech separation and enhancement task on the AVSpeech dataset show that the proposed method significantly improves the signal-to-distortion ratio over the baseline model and even outperforms an audio visual model, that utilizes visual information of lip movements.