A Reinforcement Learning Framework for Online Speaker Diarization

TL;DR

This paper introduces a pioneering reinforcement learning framework for real-time, online speaker diarization that learns to identify speakers on the fly without prior training, suitable for dynamic multi-user environments.

Contribution

It presents the first reinforcement learning-based approach for speaker diarization, integrating embedding, clustering, and resegmentation into an online decision-making process.

Findings

Effective real-time speaker identification without pretraining

Adaptive system capable of handling new speakers dynamically

Proof-of-concept desktop application demonstrating feasibility

Abstract

Speaker diarization is a task to label an audio or video recording with the identity of the speaker at each given time stamp. In this work, we propose a novel machine learning framework to conduct real-time multi-speaker diarization and recognition without prior registration and pretraining in a fully online and reinforcement learning setting. Our framework combines embedding extraction, clustering, and resegmentation into the same problem as an online decision-making problem. We discuss practical considerations and advanced techniques such as the offline reinforcement learning, semi-supervision, and domain adaptation to address the challenges of limited training data and out-of-distribution environments. Our approach considers speaker diarization as a fully online learning problem of the speaker recognition task, where the agent receives no pretraining from any training set before deployment, and learns to detect speaker identity on the fly through reward feedbacks. The paradigm of the reinforcement learning approach to speaker diarization presents an adaptive, lightweight, and generalizable system that is useful for multi-user teleconferences, where many people might come and go without extensive pre-registration ahead of time. Lastly, we provide a desktop application that uses our proposed approach as a proof of concept. To the best of our knowledge, this is the first approach to apply a reinforcement learning approach to the speaker diarization task.