SURT 2.0: Advances in Transducer-based Multi-talker Speech Recognition

TL;DR

SURT 2.0 introduces significant improvements to multi-talker speech recognition by fixing previous limitations, enhancing efficiency, and achieving state-of-the-art results on multiple benchmarks with practical training methods.

Contribution

The paper presents modifications to the original SURT model, including new unmixing, encoding, and training strategies, enabling better performance and efficiency in multi-talker ASR.

Findings

Outperforms previous SURT on multiple benchmarks

Achieves low WERs on LibriCSS, AMI, and ICSI datasets

Enables training with standard academic resources

Abstract

The Streaming Unmixing and Recognition Transducer (SURT) model was proposed recently as an end-to-end approach for continuous, streaming, multi-talker speech recognition (ASR). Despite impressive results on multi-turn meetings, SURT has notable limitations: (i) it suffers from leakage and omission related errors; (ii) it is computationally expensive, due to which it has not seen adoption in academia; and (iii) it has only been evaluated on synthetic mixtures. In this work, we propose several modifications to the original SURT which are carefully designed to fix the above limitations. In particular, we (i) change the unmixing module to a mask estimator that uses dual-path modeling, (ii) use a streaming zipformer encoder and a stateless decoder for the transducer, (iii) perform mixture simulation using force-aligned subsegments, (iv) pre-train the transducer on single-speaker data, (v) use auxiliary objectives in the form of masking loss and encoder CTC loss, and (vi) perform domain adaptation for far-field recognition. We show that our modifications allow SURT 2.0 to outperform its predecessor in terms of multi-talker ASR results, while being efficient enough to train with academic resources. We conduct our evaluations on 3 publicly available meeting benchmarks -- LibriCSS, AMI, and ICSI, where our best model achieves WERs of 16.9%, 44.6% and 32.2%, respectively, on far-field unsegmented recordings. We release training recipes and pre-trained models: https://sites.google.com/view/surt2.