Continuous Streaming Multi-Talker ASR with Dual-path Transducers

TL;DR

This paper advances streaming multi-talker automatic speech recognition by introducing dual-path models that improve performance and convergence in multi-turn meetings, evaluated on LibriCSS data.

Contribution

It proposes dual-path modeling strategies for streaming multi-talker ASR, enhancing performance and training efficiency over naive extensions of single-turn models.

Findings

Dual-path models improve word error rate (WER) in multi-turn meetings.

Training strategies like chunk width randomization and curriculum learning are crucial.

Models perform competitively with offline separation methods on LibriCSS.

Abstract

Streaming recognition of multi-talker conversations has so far been evaluated only for 2-speaker single-turn sessions. In this paper, we investigate it for multi-turn meetings containing multiple speakers using the Streaming Unmixing and Recognition Transducer (SURT) model, and show that naively extending the single-turn model to this harder setting incurs a performance penalty. As a solution, we propose the dual-path (DP) modeling strategy first used for time-domain speech separation. We experiment with LSTM and Transformer based DP models, and show that they improve word error rate (WER) performance while yielding faster convergence. We also explore training strategies such as chunk width randomization and curriculum learning for these models, and demonstrate their importance through ablation studies. Finally, we evaluate our models on the LibriCSS meeting data, where they perform competitively with offline separation-based methods.