Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition

TL;DR

This paper introduces a transformer-based model that simultaneously performs speech recognition and speaker diarization for overlapped multi-talker speech, improving accuracy in complex scenarios.

Contribution

It proposes a novel speaker mask branch within an autoregressive transformer model to enhance multi-talker speech recognition and diarization tasks.

Findings

Improved speaker diarization accuracy in overlapped speech scenarios

Effective joint speech recognition and speaker diarization with a single model

Demonstrated superior performance on LibriSpeech-based overlapped dataset

Abstract

Multi-talker overlapped speech recognition remains a significant challenge, requiring not only speech recognition but also speaker diarization tasks to be addressed. In this paper, to better address these tasks, we first introduce speaker labels into an autoregressive transformer-based speech recognition model to support multi-speaker overlapped speech recognition. Then, to improve speaker diarization, we propose a novel speaker mask branch to detection the speech segments of individual speakers. With the proposed model, we can perform both speech recognition and speaker diarization tasks simultaneously using a single model. Experimental results on the LibriSpeech-based overlapped dataset demonstrate the effectiveness of the proposed method in both speech recognition and speaker diarization tasks, particularly enhancing the accuracy of speaker diarization in relatively complex multi-talker scenarios.