Closing the Gap between Single-User and Multi-User VoiceFilter-Lite

TL;DR

This paper enhances multi-user VoiceFilter-Lite by using dual learning rates and FiLM conditioning, effectively closing the performance gap with single-user models and supporting multiple users in smart devices.

Contribution

Introduces a novel training and conditioning approach that improves multi-user VoiceFilter-Lite performance and scalability over previous methods.

Findings

Closed the performance gap between multi-user and single-user models.

Significantly improved multi-speaker evaluation results.

Supported any number of users with the new model.

Abstract

VoiceFilter-Lite is a speaker-conditioned voice separation model that plays a crucial role in improving speech recognition and speaker verification by suppressing overlapping speech from non-target speakers. However, one limitation of VoiceFilter-Lite, and other speaker-conditioned speech models in general, is that these models are usually limited to a single target speaker. This is undesirable as most smart home devices now support multiple enrolled users. In order to extend the benefits of personalization to multiple users, we previously developed an attention-based speaker selection mechanism and applied it to VoiceFilter-Lite. However, the original multi-user VoiceFilter-Lite model suffers from significant performance degradation compared with single-user models. In this paper, we devised a series of experiments to improve the multi-user VoiceFilter-Lite model. By incorporating a dual learning rate schedule and by using feature-wise linear modulation (FiLM) to condition the model with the attended speaker embedding, we successfully closed the performance gap between multi-user and single-user VoiceFilter-Lite models on single-speaker evaluations. At the same time, the new model can also be easily extended to support any number of users, and significantly outperforms our previously published model on multi-speaker evaluations.