Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings

TL;DR

This paper introduces a knowledge distillation approach for extracting short-context speaker embeddings to improve low-latency multi-speaker tracking, especially in overlapping speech scenarios.

Contribution

It proposes a novel KD-based training method for short-context speaker embeddings and explores blockwise identity reassignment for low-latency tracking.

Findings

Distilled models effectively extract short-context embeddings.

Models show increased robustness to overlapping speech.

Blockwise reassignment offers a promising low-latency tracking approach.

Abstract

Speaker embeddings are promising identity-related features that can enhance the identity assignment performance of a tracking system by leveraging its spatial predictions, i.e, by performing identity reassignment. Common speaker embedding extractors usually struggle with short temporal contexts and overlapping speech, which imposes long-term identity reassignment to exploit longer temporal contexts. However, this increases the probability of tracking system errors, which in turn impacts negatively on identity reassignment. To address this, we propose a Knowledge Distillation (KD) based training approach for short context speaker embedding extraction from two speaker mixtures. We leverage the spatial information of the speaker of interest using beamforming to reduce overlap. We study the feasibility of performing identity reassignment over blocks of fixed size, i.e., blockwise identity reassignment, to go towards a low-latency speaker embedding based tracking system. Results demonstrate that our distilled models are effective at short-context embedding extraction and more robust to overlap. Although, blockwise reassignment results indicate that further work is needed to handle simultaneous speech more effectively.