Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering

TL;DR

This paper introduces Streaming Sortformer, a real-time speaker diarization method that uses an arrival-time ordered speaker cache to improve online speaker tracking with high efficiency and accuracy.

Contribution

It proposes a novel streaming extension for Sortformer utilizing an arrival-order speaker cache for dynamic, efficient speaker embedding management in real-time diarization.

Findings

Effective in low-latency setups

Outperforms traditional methods on benchmark datasets

Robust speaker tracking in streaming scenarios

Abstract

This paper presents a streaming extension for the Sortformer speaker diarization framework, whose key property is the arrival-time ordering of output speakers. The proposed approach employs an Arrival-Order Speaker Cache (AOSC) to store frame-level acoustic embeddings of previously observed speakers. Unlike conventional speaker-tracing buffers, AOSC orders embeddings by speaker index corresponding to their arrival time order, and is dynamically updated by selecting frames with the highest scores based on the model's past predictions. Notably, the number of stored embeddings per speaker is determined dynamically by the update mechanism, ensuring efficient cache utilization and precise speaker tracking. Experiments on benchmark datasets confirm the effectiveness and flexibility of our approach, even in low-latency setups. These results establish Streaming Sortformer as a robust solution for real-time multi-speaker tracking and a foundation for streaming multi-talker speech processing.