End-to-end Music-mixed Speech Recognition

TL;DR

This paper introduces a novel end-to-end approach for improving speech recognition in multimedia by using time-domain source separation with Conv-TasNet, significantly reducing word error rates across various music genres.

Contribution

It proposes a joint fine-tuning method combining Conv-TasNet with an attention-based ASR, outperforming frequency-domain separation in mixed speech recognition tasks.

Findings

Time-domain separation drastically improves ASR performance.

Joint optimization further reduces word error rate.

Method is robust across diverse music genres.

Abstract

Automatic speech recognition (ASR) in multimedia content is one of the promising applications, but speech data in this kind of content are frequently mixed with background music, which is harmful for the performance of ASR. In this study, we propose a method for improving ASR with background music based on time-domain source separation. We utilize Conv-TasNet as a separation network, which has achieved state-of-the-art performance for multi-speaker source separation, to extract the speech signal from a speech-music mixture in the waveform domain. We also propose joint fine-tuning of a pre-trained Conv-TasNet front-end with an attention-based ASR back-end using both separation and ASR objectives. We evaluated our method through ASR experiments using speech data mixed with background music from a wide variety of Japanese animations. We show that time-domain speech-music separation drastically improves ASR performance of the back-end model trained with mixture data, and the joint optimization yielded a further significant WER reduction. The time-domain separation method outperformed a frequency-domain separation method, which reuses the phase information of the input mixture signal, both in simple cascading and joint training settings. We also demonstrate that our method works robustly for music interference from classical, jazz and popular genres.