Dynamic Recognition of Speakers for Consent Management by Contrastive Embedding Replay

TL;DR

This paper presents a novel speaker recognition system for consent management in voice assistants, capable of dynamic registration, removal, and re-registration of speakers using contrastive embedding replay, improving efficiency and performance.

Contribution

It introduces a contrastive training approach with embedding replay buffers for dynamic speaker management, addressing key challenges in consent-based voice recognition systems.

Findings

Outperforms existing methods in accuracy and efficiency.

Demonstrates effective dynamic registration and removal of speakers.

Ensures memory-efficient operation with robust speaker recognition.

Abstract

Voice assistants overhear conversations and a consent management mechanism is required. Consent management can be implemented using speaker recognition. Users that do not give consent enrol their voice and all their further recordings are discarded. Building speaker recognition-based consent management is challenging as dynamic registration, removal, and re-registration of speakers must be efficiently handled. This work proposes a consent management system addressing the aforementioned challenges. A contrastive based training is applied to learn the underlying speaker equivariance inductive bias. The contrastive features for buckets of speakers are trained a few steps into each iteration and act as replay buffers. These features are progressively selected using a multi-strided random sampler for classification. Moreover, new methods for dynamic registration using a portion of old utterances, removal, and re-registration of speakers are proposed. The results verify memory efficiency and dynamic capabilities of the proposed methods and outperform the existing approach from the literature.