Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens

TL;DR

This paper introduces a novel multi-talker speech recognition approach that uses speaker clustering tokens and hierarchical merging to improve transcription accuracy in overlapping speech scenarios.

Contribution

The paper presents a new attention-based encoder-decoder model with speaker tokens and a hierarchical merging technique, advancing multi-speaker speech recognition methods.

Findings

Achieved 55% relative error reduction on clean data.

Achieved 36% relative error reduction on noisy data.

Effective in complex 3-mix environments.

Abstract

In many real-world scenarios, such as meetings, multiple speakers are present with an unknown number of participants, and their utterances often overlap. We address these multi-speaker challenges by a novel attention-based encoder-decoder method augmented with special speaker class tokens obtained by speaker clustering. During inference, we select multiple recognition hypotheses conditioned on predicted speaker cluster tokens, and these hypotheses are merged by agglomerative hierarchical clustering (AHC) based on the normalized edit distance. The clustered hypotheses result in the multi-speaker transcriptions with the appropriate number of speakers determined by AHC. Our experiments on the LibriMix dataset demonstrate that our proposed method was particularly effective in complex 3-mix environments, achieving a 55% relative error reduction on clean data and a 36% relative error reduction on noisy data compared with conventional serialized output training.