Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speech

TL;DR

This paper introduces a speaker-conditioned speech separator trained on speaker embeddings that naturally orders speech chunks, eliminating the need for stitching, and demonstrates improved ASR performance on real conversational data.

Contribution

It presents a novel speaker-conditioned separation model using speaker embeddings from mixed signals and a data sampling strategy that enhances real conversational speech recognition.

Findings

Significant reduction in speaker-attributed WER on Hub5 data.

Model naturally orders speech chunks without stitching.

Effective generalization to real conversational speech.

Abstract

Many of the recent advances in speech separation are primarily aimed at synthetic mixtures of short audio utterances with high degrees of overlap. Most of these approaches need an additional stitching step to stitch the separated speech chunks for long form audio. Since most of the approaches involve Permutation Invariant training (PIT), the order of separated speech chunks is nondeterministic and leads to difficulty in accurately stitching homogenous speaker chunks for downstream tasks like Automatic Speech Recognition (ASR). Also, most of these models are trained with synthetic mixtures and do not generalize to real conversational data. In this paper, we propose a speaker conditioned separator trained on speaker embeddings extracted directly from the mixed signal using an over-clustering based approach. This model naturally regulates the order of the separated chunks without the need for an additional stitching step. We also introduce a data sampling strategy with real and synthetic mixtures which generalizes well to real conversation speech. With this model and data sampling technique, we show significant improvements in speaker-attributed word error rate (SA-WER) on Hub5 data.