Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition

TL;DR

Personal VAD 2.0 introduces a personalized voice activity detection system optimized for on-device speech recognition, addressing challenges of quality, streaming operation, and resource constraints with novel methods and achieving state-of-the-art results.

Contribution

The paper presents novel speaker embedding modulation, a training paradigm for enrollment-less scenarios, and architecture optimizations for resource-limited on-device VAD.

Findings

Achieves state-of-the-art performance in personalized VAD

Operates effectively in streaming and enrollment-less scenarios

Optimized for low latency and limited resources

Abstract

Personalization of on-device speech recognition (ASR) has seen explosive growth in recent years, largely due to the increasing popularity of personal assistant features on mobile devices and smart home speakers. In this work, we present Personal VAD 2.0, a personalized voice activity detector that detects the voice activity of a target speaker, as part of a streaming on-device ASR system. Although previous proof-of-concept studies have validated the effectiveness of Personal VAD, there are still several critical challenges to address before this model can be used in production: first, the quality must be satisfactory in both enrollment and enrollment-less scenarios; second, it should operate in a streaming fashion; and finally, the model size should be small enough to fit a limited latency and CPU/Memory budget. To meet the multi-faceted requirements, we propose a series of novel designs: 1) advanced speaker embedding modulation methods; 2) a new training paradigm to generalize to enrollment-less conditions; 3) architecture and runtime optimizations for latency and resource restrictions. Extensive experiments on a realistic speech recognition system demonstrated the state-of-the-art performance of our proposed method.