Modeling Overlapped Speech with Shuffles

TL;DR

This paper introduces a novel approach using shuffle products and partial order FSAs for modeling, aligning, and attributing speakers in overlapped speech, enabling single-pass multi-talker alignment.

Contribution

It presents the first algorithm for single-pass alignment of overlapped speech using shuffle products and partial order FSAs, with direct speaker attribution.

Findings

Effective alignment on synthetic LibriSpeech overlaps

First single-pass multi-talker alignment algorithm

Implementation using k2 / Icefall

Abstract

We propose to model parallel streams of data, such as overlapped speech, using shuffles. Specifically, this paper shows how the shuffle product and partial order finite-state automata (FSAs) can be used for alignment and speaker-attributed transcription of overlapped speech. We train using the total score on these FSAs as a loss function, marginalizing over all possible serializations of overlapping sequences at subword, word, and phrase levels. To reduce graph size, we impose temporal constraints by constructing partial order FSAs. We address speaker attribution by modeling (token, speaker) tuples directly. Viterbi alignment through the shuffle product FSA directly enables one-pass alignment. We evaluate performance on synthetic LibriSpeech overlaps. To our knowledge, this is the first algorithm that enables single-pass alignment of multi-talker recordings. All algorithms are implemented using k2 / Icefall.