Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speech

TL;DR

This paper introduces a novel streaming model for multi-party speech recognition and segmentation, integrating speech separation, recognition, and segmentation to improve accuracy and latency in multi-turn conversations.

Contribution

The work presents a new separator-transducer-segmenter model with innovative segmentation strategies, regularization techniques, and latency penalties for better multi-party speech processing.

Findings

Achieved 4.6% improvement in turn counting accuracy

Reduced word error rate by 17% on LibriCSS dataset

Enhanced segmentation without degrading recognition accuracy

Abstract

Streaming recognition and segmentation of multi-party conversations with overlapping speech is crucial for the next generation of voice assistant applications. In this work we address its challenges discovered in the previous work on multi-turn recurrent neural network transducer (MT-RNN-T) with a novel approach, separator-transducer-segmenter (STS), that enables tighter integration of speech separation, recognition and segmentation in a single model. First, we propose a new segmentation modeling strategy through start-of-turn and end-of-turn tokens that improves segmentation without recognition accuracy degradation. Second, we further improve both speech recognition and segmentation accuracy through an emission regularization method, FastEmit, and multi-task training with speech activity information as an additional training signal. Third, we experiment with end-of-turn emission latency penalty to improve end-point detection for each speaker turn. Finally, we establish a novel framework for segmentation analysis of multi-party conversations through emission latency metrics. With our best model, we report 4.6% abs. turn counting accuracy improvement and 17% rel. word error rate (WER) improvement on LibriCSS dataset compared to the previously published work.